Interfacing with a mobile telepresence robot

ABSTRACT

A telepresence robot may include a drive system, a control system, an imaging system, and a mapping module. The mapping module may access a plan view map of an area and tags associated with the area. In various embodiments, each tag may include tag coordinates and tag information, which may include a tag annotation. A tag identification system may identify tags within a predetermined range of the current position and the control system may execute an action based on an identified tag whose tag information comprises a telepresence robot action modifier. The telepresence robot may rotate an upper portion independent from a lower portion. A remote terminal may allow an operator to control the telepresence robot using any combination of control methods, including by selecting a destination in a live video feed, by selecting a destination on a plan view map, or by using a joystick or other peripheral device.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. patent application claims priority under 35 U.S.C. §119(e) toU.S. Provisional Application No. 61/437,433 filed on Jan. 28, 2011,which is hereby incorporated by reference in its entirety. In addition,U.S. Patent Publication No. 2007/0199108 and U.S. Pat. No. 6,535,793 arealso incorporated herein by reference in their entireties.

TECHNICAL FIELD

This disclosure relates to mobile telepresence robots.

BACKGROUND

A robot is generally an electro-mechanical machine guided by a computeror electronic programming. Telepresence robots have the capability tomove around in their environment and are not fixed to one physicallocation. An example of a telepresence robot that is in common use todayis an automated guided vehicle or automatic guided vehicle (AGV). An AGVis generally a telepresence robot that follows markers or wires in thefloor, or uses a vision system or lasers for navigation. Telepresencerobots can be found in industry, military and security environments.They also appear as consumer products, for entertainment or to performcertain tasks like home assistance.

SUMMARY

One aspect of the disclosure provides a telepresence robot systemincluding a local terminal and a remote telepresence robot. The localterminal may include an electronic display, a processor, and a memory incommunication with the processor, the memory comprising instructionsexecutable by the processor. The executable instructions may beconfigured to cause the processor to retrieve at least a portion of aplan view map representative of robot-navigable areas of a robotoperating surface; retrieve at least one of a plurality of tags, each ofthe plurality of tags comprising tag coordinates describing the relativelocation of the tag and tag information, which may include a tagannotation; receive a video feed from an imaging system of a remotetelepresence robot; receive positioning information; display the videofeed from the imaging system of the remote telepresence robot; displaythe plan view map with an indication of a current position of thetelepresence robot on the plan view map; display a rendition of the tagannotation of the at least one tag on at least one of the plan view mapand the video feed using the tag coordinates; and transmit one or morecommands to the remote telepresence robot.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to determine a distortion(e.g., a coordinate transformation between a two-dimensional coordinatesystem and a three-dimensional coordinate system) between the plan viewmap and the video feed received from the imaging system of the remotetelepresence robot; apply the distortion to the tag coordinates of theat least one tag to determine corresponding video coordinates andperspective data describing a location and perspective of the at leastone tag relative to the video feed; and display a three-dimensionalrendition of the tag annotation of the at least one tag overlaid on thevideo feed using the tag video coordinates.

In some embodiments, the three-dimensional rendition of the tagannotation may be dynamically re-rendered based on the current positionof the remote telepresence robot and a perspective of the at least onetag relative to the video feed.

In some embodiments, the three-dimensional rendition of the tagannotation may be overlaid on the video feed with respect to an objectdetected in the video feed.

In some embodiments, the three-dimensional rendition of the tagannotation may be overlaid along a wall detected in the video feed.

In some embodiments, the tag information of the at least one tagcomprises a telepresence robot action modifier and the robot actionmodifier may be configured to provide execution instructions to acontrol system of the telepresence robot to execute a first action inresponse to the telepresence robot being within a predetermined range ofthe tag coordinates of the at least one tag.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to transmit the executioninstruction to the control system of the telepresence robot when thetelepresence robot is within a predetermined range of the tagcoordinates of the at least one tag.

In some embodiments, the robot action modifier further comprisesinstructions regarding one of a time and a location on the plan view mapthat the control system of the telepresence robot should execute thefirst action.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to receive a sequence ofcoordinates relative to the plan view map forming a path along which theremote telepresence robot has traveled; store the sequence ofcoordinates forming the path as a path tag comprising tag coordinatesand tag information, which may include a tag annotation; retrieve thepath tag when the remote telepresence robot arrives within apredetermined distance of the tag coordinates; and display a renditionof the tag annotation of the path tag on at least one of the plan viewmap and the video feed using the tag coordinates.

In some embodiments, the telepresence robot system local terminalfurther comprises at least one user input device and the sequence ofcoordinates forming the path may be provided by the user input device.

In some embodiments, the sequence of coordinates forming the path may beprovided by the remote telepresence robot.

In some embodiments, the telepresence robot system further comprises acommunication system configured to facilitate communication between thetelepresence robot system local terminal and the remote telepresencerobot.

In some embodiments, the local terminal further comprises at least oneuser input device and the user input device may be configured to allow auser to provide an indication of a desired destination of the remotetelepresence robot on at least one of the plan view map and the videofeed from the imaging system of the remote telepresence robot; and thecommand transmitted to the remote telepresence robot comprises thedesired destination.

In some embodiments, the sequence of coordinates forming the robot pathmay be based at least in part on tagging information associated with theat least one tag.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to determine a sequence ofcoordinates relative to the plan view map to create a robot path betweenthe current position of the remote telepresence robot and the desireddestination of the remote telepresence robot and the command transmittedto the remote telepresence robot comprises the sequence of coordinatesforming the robot path.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to display the sequence ofcoordinates forming the robot path overlaid on the plan view map.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to determine a distortion(e.g., a coordinate transformation between a two-dimensional coordinatesystem and a three-dimensional coordinate system) between the plan viewmap and the video feed received from the imaging system of the remotetelepresence robot; apply the distortion to the sequence of coordinatesforming the robot path to determine corresponding video coordinates andperspective data describing a location and perspective of the sequenceof coordinates relative to the video feed; and display athree-dimensional rendition of the sequence of coordinates forming therobot path overlaid on the video feed.

In some embodiments, the three-dimensional rendition of the sequence ofcoordinates forming the robot path may be overlaid on the video feedwith respect to a floor detected in the video feed.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to receive a sequence ofcoordinates relative to the plan view map from a navigation system ofthe remote telepresence robot, the sequence of coordinates forming arobot path between the current position of the remote telepresence robotand a desired destination of the remote telepresence robot; and displaythe sequence of coordinates forming the robot path overlaid on the planview map.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to determine a distortion(e.g., a coordinate transformation between a two-dimensional coordinatesystem and a three-dimensional coordinate system) between the plan viewmap and the video feed received from the imaging system of the remotetelepresence robot; apply the distortion to the sequence of coordinatesforming the robot path to determine corresponding video coordinates andperspective data describing the location and perspective of the sequenceof coordinates relative to the video feed; and display athree-dimensional rendition of the sequence of coordinates forming therobot path overlaid on the video feed.

In some embodiments, the three-dimensional rendition of the sequence ofcoordinates forming the robot path may be overlaid on the video feedwith respect to a floor detected in the video feed.

In some embodiments, the tag information comprises information regardingone of: an availability of a wireless communication signal, a speed theremote telepresence robot should travel, a location of a point ofinterest, a location of a person, a location of a docking station, alocation of a rest area, a location of a glass wall, a location of aramp, a location of an object, an optimal route to navigate a tightarea, an optimal rout to navigate a congested area, and an action aremote telepresence robot should execute.

In some embodiments, the tag information may relate to a position, apath, and/or a volume, and the control system may be configured toexecute an action relative to the position, the path, and/or the volume.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to receive coordinates on theplan view map of an obstacle detected by a sensor system of the remotetelepresence robot.

In some embodiments, the plan view map and the plurality of tags arestored remotely.

In some embodiments, the plan view map and the plurality of tags arestored within the remote telepresence robot.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to determine a distortion(e.g., a coordinate transformation between a two-dimensional coordinatesystem and a three-dimensional coordinate system) between the plan viewmap and the video feed received from the imaging system of the remotetelepresence robot; and generate a hybrid map view comprising a blendedview of the plan view map and the video feed from the imaging system ofthe remote telepresence robot.

In some embodiments, the hybrid map view comprises a three-dimensionalrepresentation of the plan view map overlaid on the video feed.

In some embodiments, the telepresence robot system local terminalfurther comprises at least one user input device and the instructionsexecutable by the processor are further configured to cause theprocessor to receive a request via the at least one input device for arendered look ahead for a virtual location of the remote telepresencerobot on the plan view map; determine a distortion (e.g., a coordinatetransformation between a two-dimensional coordinate system and athree-dimensional coordinate system) between the plan view map and thevideo feed received from the imaging system of the remote telepresencerobot; and generate a virtual three-dimensional video feed based on avirtual location of the remote telepresence robot; and display thevirtual three-dimensional video feed based on the virtual location ofthe remote telepresence robot.

In some embodiments, the tag information of the at least one tagcomprises a set of coordinates with respect to the plan view mapdefining a protected region, and the tag annotation of the at least onetag may be configured to indicate the presence of a protected region.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to receive a request to createa new tag; associate tag coordinates describing a relative location ofthe new tag and tag information, which may include a tag annotation withthe new tag; and display a rendition of the tag annotation of the newtag on at least one of the plan view map and the video feed using thetag coordinates.

In some embodiments, the request to create the new tag may be generatedby the remote telepresence robot.

In some embodiments, the request to create the new tag may beautomatically generated based on a detected object in the video feed.

In some embodiments, the new tag may be a temporary tag configured toexpire once the detected object is no longer present in the video feed.

In some embodiments, the object may be a person and the tag informationof the new tag comprises identification information associated with theperson.

In some embodiments, the object may be a person and the tag informationof the new tag comprises potential actions the remote telepresence robotcan execute with respect to the person.

In some embodiments, the request to create the new tag may be generatedby a user input device in communication with the telepresence robotsystem local terminal.

In some embodiments, the request to create the new tag is made withrespect to the video feed.

In some embodiments, the request to create the new tag is made withrespect to the plan view map.

In some embodiments, the request to create a new tag is made withrespect to the current position of the remote telepresence robot.

In some embodiments, the tag information comprises information regardingone of: an availability of a wireless communication signal, a speed theremote telepresence robot should travel, a location of a point ofinterest, a location of a person, a location of a docking station, alocation of a rest area, a location of a glass wall, a location of aramp, a location of an object, an optimal route to navigate a tightarea, an optimal rout to navigate a congested area, and an action aremote telepresence robot should execute.

In other embodiments, a telepresence robot may communicate with a remoteterminal. The telepresence robot may include a drive system configuredto move the telepresence robot according to drive instructions; acontrol system in communication with the drive system, the controlsystem configured to generate drive instructions to cause the drivesystem to move the telepresence robot; an imaging system incommunication with the control system; a mapping module in communicationwith the control system, the mapping module configured to access a mapdata source, the map data source comprising a plan view maprepresentative of robot-navigable areas of a robot operating surface;and a plurality of tags, each tag being a data structure comprising tagcoordinates describing the relative location of the tag and taginformation, which may include a tag annotation; a positioning system incommunication with the control system configured to provide a currentposition with respect to the plan view map; a tag identification systemconfigured to identify at least one tag relevant to a navigation path ofthe telepresence robot; and a communication system configured tofacilitate communication between the control system and a remoteterminal, and the control system may be configured to execute an actionbased on an identified tag whose tag information comprises atelepresence robot action modifier.

In some embodiments, the tagging information for the identified tagcomprises instructions regarding one of a time and a location on theplan view map that the control system should execute the action.

In some embodiments, the control system may be configured to transmit avideo feed from the imaging system to the remote terminal via thecommunication system and the control system may be configured to receivean indication of a desired destination on the plan view map from theremote terminal via the communication system.

In some embodiments, the telepresence robot may further comprise: aplurality of sensors configured to identify obstacles in the vicinity ofthe telepresence robot and an obstacle avoidance system in communicationwith the plurality of sensors and in communication with the controlsystem, where the control system may be further configured to generateadditional drive instructions to avoid obstacles in the vicinity of thetelepresence robot.

In some embodiments, the plurality of sensors comprises at least one ofa proximity sensor, a contact sensor, an odometry sensor, and athree-dimensional image sensor.

In some embodiments, the plurality of sensors may comprise athree-dimensional image sensor that forms a point cloud, including athree-dimensional occupancy of obstacles, and the drive instructions maybe configured to avoid the three-dimensional occupancy of the obstacles.

In some embodiments, the telepresence robot may further comprise: a mapgeneration system in communication with the control system, the mapgeneration system configured to autonomously create the plan view map ofthe robot operating surface, where the control system generates driveinstructions to cause the telepresence robot to move throughout therobot operating surface and obtain a plurality of measurements, and themap generation system uses the plurality of measurements to generate theplan view map.

In some embodiments, the telepresence robot may further comprise anavigation system configured to generate a navigation path comprising asequence of coordinates from the current position on the plan view mapto the desired destination on the plan view map.

In some embodiments, the telepresence robot may transmit coordinatesrelative to the plan view map of a detected obstacle to the remoteterminal via the communication system.

In some embodiments, the sequence of coordinates forming the navigationpath may be based at least in part on tagging information associatedwith the identified tag.

In some embodiments, the navigation system is configured to generate thenavigation path by selecting a navigation path from a plurality ofpotential navigation paths, and the tags relevant to the navigation pathof the telepresence robot are associated with the plurality of potentialnavigation paths, and the navigation system is configured to select thenavigation path based at least in part on the identified relevant tags.

In some embodiments, the sequence of coordinates forming the navigationpath is transmitted via the communication system to the remote terminal.

In some embodiments, the telepresence robot may be configured to createa new tag using the sequence of coordinates forming the navigation path,such that the new tag comprises the sequence of coordinates, tagginginformation related to the navigation path, and a tag annotation relatedto the navigation path.

In some embodiments, the tag information of each of the plurality oftags comprises information regarding one of: an availability of awireless communication signal, a speed the remote telepresence robotshould travel, a location of a point of interest, a location of aperson, a location of a docking station, a location of a rest area, alocation of a glass wall, a location of a ramp, a location of an object,an optimal route to navigate a tight area, an optimal rout to navigate acongested area, and an action a remote telepresence robot shouldexecute.

In some embodiments, the control system may be further configured toreceive a navigation path from the current position on the plan view mapto the desired destination on the plan view map and the control systemmay be further configured to generate drive instructions to cause thedrive system to move the telepresence robot to the desired destinationbased on the navigation path.

In some embodiments, the communication system may be configured todetect a disruption in communication between the telepresence robot anda remote terminal, wherein the control system may be further configuredto continue to generate drive instructions to cause the telepresencerobot to autonomously move to the desired destination during thedisruption in communication.

In some embodiments, the map data source may be stored remotely, suchthat the mapping module may be configured to access the map data sourcevia the communication system.

In some embodiments, the map data source may be stored within thetelepresence robot, such that the mapping module may be configured toaccess an internal map data source.

In some embodiments, the internal map data source may be synced with aremotely stored map data source.

In some embodiments, the positioning system may be further configured toprovide a robot pose relative to the plan view map.

In some embodiments, the telepresence robot may be configured to createa new tag by: associating tag coordinates describing the relativelocation of the new tag with respect to one of the plan view map and avideo feed generated by the imaging system; associating tag informationwith the new tag; and associating a tag annotation with the new tag.

In some embodiments, the new tag may be created in response to thetelepresence robot detecting an object in the video feed.

In some embodiments, the object may be a person and the tag informationof the new tag comprises identification information associated with theperson.

In some embodiments, the object may be a person and the tag informationof the new tag comprises potential actions the remote telepresence robotcan execute with respect to the person.

In some embodiments, the tag information comprises information regardingone of: an availability of a wireless communication signal, a speed theremote telepresence robot should travel, a location of a point ofinterest, a location of a person, a location of a docking station, alocation of a rest area, a location of a glass wall, a location of aramp, a location of an object, an optimal route to navigate a tightarea, an optimal rout to navigate a congested area, and an action aremote telepresence robot should execute.

In some embodiments, the telepresence robot system may further comprise:an RFID reader in communication with the positioning system, where thepositioning system associates a plurality of RFID chips with acorresponding plurality of coordinates on the plan view map, and thepositioning system may be configured to determine the current positionof the telepresence robot based at least in part on the location of oneor more RFID chips within range of the RFID reader.

Various methods of control may be employed in the present systems andmethods. For example, a telepresence robot system local terminal maycomprise: an electronic display; a processor in communication with theelectronic display interface; a memory in communication with theprocessor, the memory comprising instructions executable by theprocessor configured to cause the processor to: retrieve at least aportion of a plan view map representative of robot-navigable areas of arobot operating surface; receive a video feed from an imaging system ofthe remote telepresence robot at a first perspective; receive a currentposition from a positioning system of the remote telepresence robot withrespect to a plan view map; display the video feed from the imagingsystem of the remote telepresence robot; display the plan view map withan indication of the current position of the telepresence robot on theplan view map; transmit a command to the remote telepresence robot; anda user input device in communication with the processor, the user inputdevice configured to allow a user to select a movement for a remotetelepresence robot, the selection of the movement comprising selecting adestination of the remote telepresence robot with respect to the videofeed; with respect to the plan view map; and by incrementally advancingthe remote telepresence robot in one of at least four possibledirections relative to the current position of the remote telepresencerobot.

In some embodiments, the selection of the movement comprises selectingan alternative perspective of the video feed by selecting a point withinthe video feed. This mode would likely be used for intermediatedistances to get to locations within view on the video feed.

In some embodiments, selection of the movement comprises selecting analternative perspective of the video feed by selecting a point on theplan view map. This mode would likely be used for farther distances(e.g., down hallways, between rooms, etc.) to locations not within viewon the video feed. In some embodiments, selection of the movementcomprises using a joystick or meta joystick in manual control. This modewould likely be used for micro/finer adjustments, e.g., within a room inclose proximity to humans/patients.

In some embodiments, the selection of the movement comprises selectingan alternative perspective of the video feed by incrementally panning ortilting the imaging system while the remote telepresence robot remainsin the current position.

In some embodiments, wherein the selection of the movement may relate torotating one of a lower portion of the remote telepresence robot and anupper portion of the remote telepresence robot.

In some embodiments, there will be a way to switch between modes, e.g.multi-modal user interface wherein one can select to control eitherhead/imaging system movement or movement of the base/lower portion ofthe remote presence robot.

In some embodiments when control of head/imaging system movement isselected there may be options to select either position-based box-zoomhead motion via mouse or velocity-based head motion via mouse.

In some embodiments when control of base/lower portion of the remotepresence robot is selected there may be options to select from one ofthe following: (1) click-on-map, i.e. top down map view and click ontarget destination or select from destination list; (2) click-on-video,i.e. position-based control that enables click on location in the videoand robot drives there; (3) joystick or meta joystick, e.g., mousevelocity-based control or arrows specifying forward, left, right, etc.

In some embodiments the functionality/information needed to be accessedby user at all times while robot base is moving includes: (1) remoteview, i.e., where the robot is headed (view should be large enough toprovide meaningful visual information for user to operate safely); (2)for supervisory control modes, potential need for override capability tocancel/abort operation as needed.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to receive a selection of adestination of the remote robot from the user input device; determine asequence of coordinates relative to the plan view map to create anavigation path between the current position of the remote telepresencerobot and the selected destination of the remote telepresence robot; andtransmit a command to the remote telepresence robot comprising thesequence of coordinates forming the navigation path.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to display the sequence ofcoordinates forming the navigation path overlaid on the plan view map.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to: determine a distortionbetween the plan view map and the video feed received from the imagingsystem of the remote telepresence robot (e.g., a coordinatetransformation between a two-dimensional coordinate system and athree-dimensional coordinate system); apply the distortion to thesequence of coordinates forming the navigation path to determinecorresponding video coordinates and perspective data describing thelocation and perspective of the sequence of coordinates relative to thevideo feed; and display a three-dimensional rendition of the sequence ofcoordinates forming the navigation path overlaid on the video feed.

In some embodiments, the three-dimensional rendition of the sequence ofcoordinates forming the navigation path may be overlaid on the videofeed with respect to a floor detected in the video feed.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to receive a selection of adestination of the remote robot from the user input device; transmitdestination coordinates relative to the plan view map to the remotetelepresence robot, the destination coordinates corresponding to theselected destination; receive a sequence of coordinates relative to theplan view map from a navigation system of the remote telepresence robot,the sequence of coordinates forming a navigation path between thecurrent position of the remote telepresence robot and the desireddestination of the remote telepresence robot; and display the sequenceof coordinates forming the navigation path overlaid on the plan viewmap.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to determine a distortionbetween the plan view map and the video feed received from the imagingsystem of the remote telepresence robot (e.g., a coordinatetransformation between a two-dimensional coordinate system and athree-dimensional coordinate system); apply the distortion to thesequence of coordinates forming the navigation path to determinecorresponding video coordinates and perspective data describing thelocation and perspective of the sequence of coordinates relative to thevideo feed; and display a three-dimensional rendition of the sequence ofcoordinates forming the navigation path overlaid on the video feed.

In some embodiments, the three-dimensional rendition of the sequence ofcoordinates forming the navigation path may be overlaid on the videofeed with respect to a floor detected in the video feed.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to receive coordinates on theplan view map of an obstacle detected by a sensor system of the remotetelepresence robot.

In some embodiments, the plan view map is stored remotely.

In some embodiments, the plan view map is stored within the remotetelepresence robot.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to determine a distortionbetween the plan view map and the video feed received from the imagingsystem of the remote telepresence robot (e.g., a coordinatetransformation between a two-dimensional coordinate system and athree-dimensional coordinate system); and generate a hybrid map viewcomprising a blended view of the plan view map and the video feed fromthe imaging system of the remote telepresence robot.

In some embodiments, the hybrid map view comprises a three-dimensionalrepresentation of the plan view map overlaid on the video feed.

In some embodiments, the instructions executable by the processor arefurther configured to cause the processor to receive a request via theinput device for a rendered look-ahead for a virtual location of theremote telepresence robot on the plan view map determine a distortionbetween the plan view map and the video feed received from the imagingsystem of the remote telepresence robot (e.g., a coordinatetransformation between a two-dimensional coordinate system and athree-dimensional coordinate system); and generate a virtualthree-dimensional video feed based on a virtual location of the remotetelepresence robot; and display the virtual three-dimensional video feedbased on the virtual location of the remote telepresence robot.

In some embodiments, a robot may be configured to unwind and/or tocontrol an upper portion and a lower portion independently in order toappear human-like. For example, a telepresence robot may comprise: anupper portion; a lower portion rotatably connected to the upper portion;a drive system configured to move the telepresence robot according todrive instructions; a control system in communication with the drivesystem, the control system configured to generate drive instructions tocause the drive system to move the telepresence robot; a rotation systemconfigured to rotate the robot from a first heading to a second headingby rotating the upper portion and the lower portion independently.

In some embodiments, the rotation system may be configured to rotate therobot toward a second heading by rotating the upper portion of the robottoward the second heading; detecting that the upper portion of the robothas reached a panning limit of the upper portion of the robot relativeto the lower portion of the robot; begin rotating the lower portion ofthe robot toward the second heading at the panning limit of the upperportion of the robot; detect that the upper portion of the robot hasreached the second heading; and continue rotating the lower portion ofthe robot toward the second heading while simultaneouslycounter-rotating the upper portion of the robot, such that the upperportion of the robot maintains the second heading.

In some embodiments, the panning limit may be reached when the upperportion cannot physically rotate anymore with respect to the lowerportion of the robot.

In some embodiments, the panning limit may be reached when the upperportion is misaligned with respect to the lower portion a predefinednumber of rotation degrees.

In some embodiments, the panning limit may be a function of the numberof degrees the upper portion is misaligned with respect to the lowerportion and the length of time the upper portion has been misalignedwith respect to the lower portion.

In some embodiments, the rotation system may be configured to rotate therobot toward a second heading by rotating the upper portion of the robottoward the second heading at a first rotational velocity; rotating thelower portion of the robot toward the second heading at a secondrotation velocity; detect that the upper portion of the robot hasreached the second heading; and continue rotating the lower portion ofthe robot toward the second heading while simultaneouslycounter-rotating the upper portion of the robot, such that the upperportion of the robot maintains the second heading.

In some embodiments the telepresence robot may further comprise animaging system in communication with the control system and apositioning system in communication with the control system configuredto provide a current position of the robot relative to a plan view mapand a current alignment of the upper portion with respect to the planview map, where the control system may be configured to transmit a videofeed from the imaging system, the current position of the robot, and thecurrent alignment of the upper portion to a remote terminal, such thatthe remote terminal can determine a distortion between the plan view mapand the video feed received from the imaging system of the remotetelepresence robot (e.g., a coordinate transformation between atwo-dimensional coordinate system and a three-dimensional coordinatesystem); apply the distortion to a tag having coordinates associatedwith the plan view map in order determine corresponding videocoordinates and perspective data describing the location and perspectiveof the tag relative to the video feed; and display a three-dimensionalrendition of the tag overlaid on the video feed using the videocoordinates.

The above described embodiments are described from the perspective of arobot and/or a local terminal. It should be apparent to one of skill inthe art that the embodiments described above could be implemented assystems, adapted as methods performed by a system, or embodied in acomputer-readable medium that could be executed by a system. Forexample, a method for changing a heading of a robot may comprisetransmitting a heading to a control system of a robot, the controlsystem of the robot in communication with a drive system configured tomove the robot according to drive instructions and rotating an upperportion of the robot toward the heading independently from the lowerportion of the robot.

In some embodiments, a method for controlling a remote telepresencerobot may comprise retrieving at least a portion of a plan view maprepresentative of robot-navigable areas of a robot operating surface;retrieving at least one of a plurality of tags, each of the plurality oftags comprising tag coordinates describing the relative location of thetag and tag information; receiving a video feed from an imaging systemof a remote telepresence robot; receiving positioning informationassociated with a current position of the remote telepresence robot;displaying, via an electronic display, the video feed from the imagingsystem of the remote telepresence robot; displaying, via the electronicdisplay, a rendition of the tag information of the at least one tag onthe video feed using the tag coordinates; and transmitting a command tothe remote telepresence robot. A method for controlling a telepresencerobot, comprising retrieving at least a portion of a plan view map;retrieving at least one of a plurality of tags, each tag being a datastructure comprising tag coordinates describing the relative location ofthe tag and tag information; determining a current position relative tothe plan view map; identifying at least one tag of the plurality of tagsrelevant to a navigation path of the telepresence robot; executing anaction based on the identified tag whose tag information comprises atelepresence action modifier.

In some embodiments, a method for controlling a telepresence robot maycomprise retrieving at least a portion of a plan view map representativeof robot-navigable areas of a robot operating surface; receiving a videofeed from an imaging system of the remote telepresence robot at a firstperspective; receiving positioning data associated with a currentposition of the remote telepresence robot; displaying the video feedfrom the imaging system of the remote telepresence robot; andtransmitting a command to the remote telepresence robot; and receiving aplurality of movement selections from a user input device movement, themovement selections made (1) with respect to the video feed; (2) withrespect to the plan view map; and/or (3) by incrementally advancing theremote telepresence robot in a direction relative to the currentposition of the remote telepresence robot.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a perspective view of an exemplary telepresence robot.

FIG. 2 is an elevated perspective view of an exemplary telepresencerobot.

FIGS. 3A-3C are schematic views of exemplary telepresence robots.

FIG. 4A is a front perspective view of an exemplary base for a mobilehuman interface robot.

FIG. 4B is a rear perspective view of the base shown in FIG. 4A.

FIG. 4C is a top view of the base shown in FIG. 4A.

FIG. 4D is a top schematic view of an exemplary base for a telepresencerobot.

FIG. 4E is a bottom perspective view of an exemplary drive system for atelepresence robot.

FIG. 4F is a top perspective view of the drive system shown in FIG. 4E.

FIG. 5 is a schematic view of an exemplary control system executed by acontroller of a telepresence robot.

FIG. 6A provides a schematic view of an exemplary robot system includingmultiple robots in communication with robot endpoint servers.

FIG. 6B illustrates a teleoperation software application executed by arobot or a terminal.

FIG. 6C illustrates one embodiment of a screen shot of a user interfacefor controlling navigation of a semi-autonomous telepresence robot.

FIG. 6D illustrates a screen shot, in which the relative area of thescreen devoted to the map window is increased.

FIG. 7 is a schematic view of an exemplary robot system architecture.

FIG. 8A is a schematic view of an exemplary occupancy map.

FIG. 8B is a schematic view of a telepresence robot having a field ofview of a scene in a working area.

FIG. 8C is a schematic view of an exemplary layout map.

FIG. 8D is a schematic view of an exemplary robot map corresponding tothe layout map shown in FIG. 8C.

FIG. 8E provides an exemplary arrangement of operations for operating atelepresence robot to navigate about an environment using a layout mapand a robot map.

FIG. 8F illustrates a method for using a robot location and perspectiveto determine a distortion between a video feed and a plan view map.

FIG. 9A is schematic view of an exemplary remote video view from a robotpositioned in a hallway.

FIG. 9B is a schematic view of an exemplary hybrid map incorporating theremote video view shown in FIG. 9A, together with a map indicating roomnumbers.

FIG. 10A provides an exemplary remote view of a remote video window of atelepresence software application.

FIG. 10B is a schematic view of an exemplary map of the area shown bythe remove view of FIG. 10A.

FIG. 10C is a schematic view of an exemplary look-ahead view of atelepresence software application.

FIG. 10D is a schematic view of the map shown in FIG. 10B with a roboticon and a corresponding camera field of view.

FIG. 10E is a schematic view of an exemplary look-ahead view of atelepresence software application.

FIG. 10F is a schematic view of the map shown in FIG. 10B with a roboticon and a corresponding camera field of view.

FIG. 10G provides an exemplary arrangement of operations for alook-ahead routine of a telepresence software application.

FIG. 11A is a schematic view of an exemplary user interface that allowsa user to specify a robot destination within an identified navigablearea.

FIG. 11B provides an exemplary arrangement of operations for a method ofnavigating a robot to a destination.

FIG. 11C is a schematic view of an exemplary user interface prompting auser that a ramp was selected as a robot destination.

FIG. 11D is a schematic view of an exemplary user interface prompting auser that an obstacle was selected as a robot destination.

FIG. 12 is a schematic view of an exemplary user interface that allows auser to specify a robot drive path within an identified navigable area.

FIG. 13 is a schematic view of an exemplary user interface thatincorporates hyper-tags and context sensitive commands.

FIG. 14 is a perspective view of an exemplary telepresence robotmaintaining a sensor field of view on a person.

FIG. 15A is a schematic view of an exemplary three-dimensional map viewthat includes hyper-tags.

FIG. 15B is a schematic view of an exemplary two-dimensional map viewthat include hyper-tags.

FIG. 16A is a schematic view of an exemplary robot system.

FIG. 16B is a schematic view of exemplary interactions with a map datasource.

FIG. 16C is a schematic view of exemplary interactions between a robotcontrol system and a map data source.

FIG. 16D is a schematic view of an exemplary robot system.

FIG. 17 is a schematic view of an exemplary user interface that includesan augmented overlay corresponding to a telepresence robot.

FIG. 18 is a schematic view of an exemplary sequence of robot actions.

FIG. 19 is a schematic view of an exemplary user interface having ascreen indicator overlaid on a remote video feed received from atelepresence robot.

FIGS. 20A-20C provide an exemplary arrangement of operations forrecovering from a loss of robot communications.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Telepresence robots can interact or interface with humans to provide anumber of services, such as a physician or healthcare worker providingremote medical consultation, home assistance, commercial assistance, andmore. In the example of home assistance, a telepresence robot can assistelderly people with everyday tasks, including, but not limited to,maintaining a medication regime, mobility assistance, communicationassistance (e.g., video conferencing, telecommunications, Internetaccess, etc.), home or site monitoring (inside and/or outside), personmonitoring, and/or providing a personal emergency response system(PERS). For commercial assistance, the telepresence robot can providevideoconferencing (e.g., in a hospital setting), a point of saleterminal, an interactive information/marketing terminal, etc.

Referring to FIGS. 1-3B, in some implementations, a telepresence robot100 includes a robot body 110 (or chassis) that defines a forward drivedirection F. The robot 100 also includes a drive system 200 (FIG. 4D),an interfacing module 300, and a sensor system 400, each supported bythe robot body 110 and in communication with a controller 500 (FIG. 5)that coordinates operation and movement of the robot 100. A power source105 (e.g., battery or batteries) can be carried by the robot body 110and in electrical communication with, and deliver power to, each ofthese components, as necessary.

The robot body 110, in the examples shown, includes a base 120, at leastone leg 130 extending upwardly from the base 120, and a torso 140supported by the at least one leg 130. The base 120 may support thedrive system 200. The robot body (lower portion) 110 also includes aneck 150 supported by the torso 140. The neck 150 supports a head (upperportion) 160, which supports at least a portion of the interfacingmodule 300. The base 120 includes enough weight (e.g., by supporting thepower source 105 (batteries) to maintain a low center of gravity CG_(B)of the base 120 and a low overall center of gravity CG_(R) of the robot100 for maintaining mechanical stability.

Referring to FIGS. 2 and 4A-4C, in some implementations, the base 120defines a trilaterally symmetric shape (e.g., a triangular shape fromthe top view). For example, the base 120 may include a base chassis 122that supports a base body 124 having first, second, and third base bodyportions 124 a, 124 b, 124 c corresponding to each leg of thetrilaterally shaped base 120 (see e.g., FIG. 4A). Each base body portion124 a, 124 b, 124 c can be movably supported by the base chassis 122 soas to move independently with respect to the base chassis 122 inresponse to contact with an object. The trilaterally symmetric shape ofthe base 120 allows bump detection 360° around the robot 100. Each basebody portion 124 a, 124 b, 124 c can have an associated contact sensor(e.g., capacitive sensor, read switch, etc.) that detects movement ofthe corresponding base body portion 124 a, 124 b, 124 c with respect tothe base chassis 122.

In some implementations, the drive system 200 provides omni-directionaland/or holonomic motion control of the robot 100. As used herein theterm “omni-directional” refers to the ability to move in substantiallyany planar direction, i.e., side-to-side (lateral), forward/back, androtational. These directions are generally referred to herein as x, y,and θz, respectively. Furthermore, the term “holonomic” is used in amanner substantially consistent with the literature use of the term andrefers to the ability to move in a planar direction with three planardegrees of freedom, i.e., two translations and one rotation. Hence, aholonomic robot has the ability to move in a planar direction at avelocity made up of substantially any proportion of the three planarvelocities (lateral, forward/back, and rotational), as well as theability to change these proportions in a substantially continuousmanner.

The robot 100 can operate in human environments (e.g., environmentstypically designed for bipedal, walking occupants) using wheeledmobility. In some implementations, the drive system 200 includes first,second, and third drive wheels 210 a, 210 b, 210 c equally spaced (i.e.,trilaterally symmetric) about the vertical axis Z (e.g., 120 degreesapart); however, other arrangements are possible as well. Referring toFIG. 4D, the drive wheels 210 a, 210 b, 210 c may define a transversearcuate rolling surface (i.e., a curved profile in a directiontransverse or perpendicular to the rolling direction D_(R)), which mayaid maneuverability of the holonomic drive system 200. Each drive wheel210 a, 210 b, 210 c is coupled to a respective drive motor 220 a, 220 b,220 c that can drive the drive wheel 210 a, 210 b, 210 c in forwardand/or reverse directions independently of the other drive motors 220 a,220 b, 220 c. Each drive motor 220 a-c can have a respective encoder,which provides wheel rotation feedback to the controller 500. In someexamples, each drive wheel 210 a, 210 b, 210 c is mounted on or near oneof the three points of an equilateral triangle and has a drive direction(forward and reverse directions) that is perpendicular to an anglebisector of the respective triangle end. Driving the trilaterallysymmetric holonomic base 120 with a forward driving direction F allowsthe robot 100 to transition into non-forward drive directions forautonomous escape from confinement or clutter and then rotate and/ortranslate to drive along the forward drive direction F after the escapehas been resolved.

Referring to FIGS. 4E and 4F, in some implementations, the drive system200 includes first, second, third, and fourth drive wheels 210 a-darranged in a square or rectangular configuration (e.g., equidistantlyfrom the Z axis) from a top view. The drive system 200 may operate in aholonomic manner, allowing strafing. Each drive wheel 210 a-d is coupledto a respective drive motor 220 a-d that can drive the drive wheel 210a-d in forward and/or reverse directions independently of the otherdrive motors 220 a-d. Each drive motor 220 a-d can have a respectiveencoder, which provides wheel rotation feedback to the controller 500. Abase chassis 122 supports the drive motors 220 a-d and thecorrespondingly coupled drive wheels 210 a-d.

In some examples, as illustrated in FIG. 2, the first drive wheel 210 ais arranged as a leading drive wheel along the forward drive direction Fwith the remaining two drive wheels 210 b, 210 c trailing behind. Inthis arrangement, to drive forward, the controller 500 may issue a drivecommand that causes the second and third drive wheels 210 b, 210 c todrive in a forward rolling direction at an equal rate while the firstdrive wheel 210 a slips along the forward drive direction F. Moreover,this drive wheel arrangement allows the robot 100 to stop short (e.g.,incur a rapid negative acceleration against the forward drive directionF). This is due to the natural dynamic instability of the three-wheeleddesign. If the forward drive direction F were along an angle bisectorbetween two forward drive wheels, stopping short would create a torquethat would force the robot 100 to fall, pivoting over its two “front”wheels. Instead, traveling with one drive wheel 210 a forward naturallysupports or prevents the robot 100 from toppling over forward, if thereis need to come to a quick stop. When accelerating from a stop, however,the controller 500 may take into account a moment of inertia I of therobot 100 from its overall center of gravity CG_(R).

In some implementations of the drive system 200, each drive wheel 210 a,210 b, 210 has a rolling direction D_(R) radially aligned with avertical axis Z, which is orthogonal to X and Y axes of the robot 100.The first drive wheel 210 a can be arranged as a leading drive wheelalong the forward drive direction F with the remaining two drive wheels210 b, 210 c trailing behind. In this arrangement, to drive forward, thecontroller 500 may issue a drive command that causes the first drivewheel 210 a to drive in a forward rolling direction and the second andthird drive wheels 210 b, 210 c to drive at an equal rate as the firstdrive wheel 210 a, but in a reverse direction.

In other implementations, the drive system 200 can be arranged to havethe first and second drive wheels 210 a, 210 b positioned such that anangle bisector of an angle between the two drive wheels 210 a, 210 b isaligned with the forward drive direction F of the robot 100. In thisarrangement, to drive forward, the controller 500 may issue a drivecommand that causes the first and second drive wheels 210 a, 210 b todrive in a forward rolling direction and an equal rate, while the thirddrive wheel 210 c drives in a reverse direction or remains idle and isdragged behind the first and second drive wheels 210 a, 210 b. To turnleft or right while driving forward, the controller 500 may issue acommand that causes the corresponding first or second drive wheel 210 a,210 b to drive at a relatively quicker/slower rate. Other drive systemarrangements can be used as well. The drive wheels 210 a, 210 b, 210 cmay define a cylindrical, circular, elliptical, or polygonal profile.

Referring again to FIGS. 1-3B, the base 120 supports at least one leg130 extending upward in the Z direction from the base 120. The leg(s)130 may be configured to have a variable height for raising and loweringthe torso 140 with respect to the base 120. In some implementations,each leg 130 includes first and second leg portions 132, 134 that movewith respect to each other (e.g., telescopic, linear, and/or angularmovement). Rather than having extrusions of successively smallerdiameter telescopically moving in and out of each other and out of arelatively larger base extrusion, the second leg portion 134, in theexamples shown, moves telescopically over the first leg portion 132,thus allowing other components to be placed along the second leg portion134 and potentially move with the second leg portion 134 to a relativelyclose proximity of the base 120. The leg 130 may include an actuatorassembly for moving the second leg portion 134 with respect to the firstleg portion 132. The actuator assembly 136 may include a motor driver incommunication with a lift motor and an encoder, which provides positionfeedback to the controller.

Generally, telescopic arrangements include successively smaller diameterextrusions telescopically moving up and out of relatively largerextrusions at the base 120 in order to keep a center of gravity CG_(L)of the entire leg 130 as low as possible. Moreover, stronger and/orlarger components can be placed at the bottom to deal with the greatertorques experienced at the base 120 when the leg 130 is fully extended.This approach, however, offers two problems. First, when the relativelysmaller components are placed at the top of the leg 130, any rain, dust,or other particulate tends to run or fall down the extrusions,infiltrating a space between the extrusions, thus obstructing nesting ofthe extrusions. This creates a very difficult sealing problem whilestill trying to maintain full mobility/articulation of the leg 130.Second, it may be desirable to mount payloads or accessories on therobot 100. One common place to mount accessories is at the top of thetorso 140. If the second leg portion 134 moves telescopically in and outof the first leg portion, accessories and components could only bemounted above the entire second leg portion 134, if they need to movewith the torso 140. Otherwise, any components mounted on the second legportion 134 would limit the telescopic movement of the leg 130.

By having the second leg portion 134 move telescopically over the firstleg portion 132, the second leg portion 134 provides additional payloadattachment points that can move vertically with respect to the base 120.This type of arrangement causes water or airborne particulate to rundown the torso 140 on the outside of every leg portion 132, 134 (e.g.,extrusion) without entering a space between the leg portions 132, 134.This greatly simplifies sealing any joints of the leg 130. Moreover,payload/accessory mounting features of the torso 140 and/or second legportion 134 are always exposed and available no matter how the leg 130is extended.

Referring to FIG. 2, the leg 130 supports the torso 140, which may havea shoulder 142 extending over and above the base 120. In the exampleshown, the torso 140 has a downward facing or bottom surface 144 (e.g.,toward the base) forming at least part of the shoulder 142 and anopposite upward facing or top surface 146 (FIG. 3A), with a side surface148 extending therebetween. The torso 140 may define various shapes orgeometries, such as a circular or an elliptical shape having a centralportion 141 supported by the leg(s) 130 and a peripheral free portion143 that extends laterally beyond a lateral extent of the leg(s) 130,thus providing an overhanging portion that defines the downward facingsurface 144. In some examples, the torso 140 defines a polygonal orother complex shape that defines a shoulder, which provides anoverhanging portion that extends beyond the leg(s) 130 over the base120.

The robot 100 may include one or more accessory ports 170 (e.g.,mechanical and/or electrical interconnect points) for receivingpayloads. The accessory ports 170 can be located so that receivedpayloads do not occlude or obstruct sensors of the sensor system 400(e.g., on the bottom and/or top surfaces 144, 146 of the torso 140,etc.).

An external surface of the torso 140 may be sensitive to contact ortouching by a user, so as to receive touch commands from the user. Forexample, when the user touches the top surface 146 of the torso 140, therobot 100 responds by lowering a height of the torso with respect to thefloor (e.g., by decreasing the height of the leg(s) 130 supporting thetorso 140). Similarly, when the user touches the bottom surface 144 ofthe torso 140, the robot 100 responds by raising the torso 140 withrespect to the floor (e.g., by increasing the height of the leg(s) 130supporting the torso 140). Moreover, upon receiving a user touch onforward, rearward, right or left portions of side surface 148 of thetorso 140, the robot 100 responds by moving in a corresponding directionof the received touch command (e.g., rearward, forward, left, and right,respectively). The external surface(s) of the torso 140 may include acapacitive sensor in communication with the controller that detects usercontact.

Referring again to FIGS. 1-3B, the torso 140 supports the neck 150,which provides panning and tilting of the head 160 with respect to thetorso 140. In the examples shown, the neck 150 includes a rotator 152and a tilter 154. The rotator 152 may provide a range of angularmovement θ_(R) (e.g., about the Z axis) of between about 90° and about360°. Other ranges are possible as well. Moreover, in some examples, therotator 152 includes electrical connectors or contacts that allowcontinuous 360° rotation of the head 160 with respect to the torso 140in an unlimited number of rotations while maintaining electricalcommunication between the head 160 and the remainder of the robot 100.The tilter 154 may include the same or similar electrical connectors orcontacts allow rotation of the head 160 with respect to the torso 140while maintaining electrical communication between the head 160 and theremainder of the robot 100. The rotator 152 may include a rotator motorcoupled to or engaging a ring (e.g., a toothed ring rack). The tilter154 may move the head at an angle θ_(T) (e.g., about the Y axis) withrespect to the torso 140 independently of the rotator 152. In someexamples that tilter 154 includes a tilter motor, which moves the head160 between an angle θ_(T) of ±90° with respect to Z axis. Other rangesare possible as well, such as ±45°, etc. The robot 100 may be configuredso that the leg(s) 130, the torso 140, the neck 150, and the head 160stay within a perimeter of the base 120 for maintaining stable mobilityof the robot 100.

The head 160 may be sensitive to contact or touching by a user, so as toreceive touch commands from the user. For example, when the user pullsthe head 160 forward, the head 160 tilts forward with passive resistanceand then holds the position. Moreover, if the user pushes/pulls the head160 vertically downward, the torso 140 may lower (via a reduction inlength of the leg 130) the head 160. The head 160 and/or neck 150 mayinclude strain gauges and/or contact sensors 165 that sense user contactor manipulation.

In some implementations, the head 160 supports one or more portions ofthe interfacing module 300. The head 160 may include a dock 302 forreleasably receiving one or more computing tablets 310, also referred toas a web pad or a tablet PC, each of which may have a touch screen 312.The web pad 310 may be oriented forward, rearward or upward. In someimplementations, web pad 310 includes a touch screen, optional I/O(e.g., buttons and/or connectors, such as micro-USB, etc.) a processor,and memory in communication with the processor. An exemplary web pad 310includes the Apple iPad by Apple, Inc. In some examples, the web pad 310functions as the controller 500 or assist the controller 500 incontrolling the robot 100. The touch screen may detect, monitor, and/orreproduce points of user touching thereon for receiving user inputs andproviding a graphical user interface that is touch interactive. In someexamples, the web pad 310 includes a touch screen caller that allows theuser to find it when it has been removed from the robot 100.

The interfacing module 300 may include a camera 320 disposed on the head160 (see e.g., FIG. 3A), which can be used to capture video from anelevated vantage point of the head 160 (e.g., for videoconferencing). Inthe example shown in FIG. 2, the camera 320 is disposed on the neck 150.In some examples, the camera 320 is operated only when the web pad 310is detached or undocked from the head 160. When the web pad 310 isattached or docked on the head 160 in the dock 302 (and optionallycovering the camera 320), the robot 100 may use a camera of the web pad310 for capturing video. In such instances, the camera 320 may bedisposed behind the docked web pad 310 and enter an active state whenthe web pad 310 is detached or undocked from the head 160 and aninactive state when the web pad 310 is attached or docked on the head160.

The robot 100 can provide videoconferencing (e.g., at 24 fps or higher)through the interface module 300 (e.g., using a web pad 310, the camera320, the microphones 330, and/or the speakers 340). Thevideoconferencing can be multiparty. The robot 100 can provide eyecontact between both parties of the videoconferencing by maneuvering thehead 160 to face the user. Moreover, the robot 100 can have a gaze angleof <5° (e.g., an angle away from an axis normal to the forward face ofthe head 160). At least one three-dimensional image sensor 450 and/orthe camera 320 on the robot 100 can capture life-size images includingbody language. The controller 500 can synchronize audio and video (e.g.,with a difference of <50 ms). The camera 320 may be movable within atleast 1° of freedom separately from the web pad 310. The head 160 mayinclude one or more speakers 340 so as to have sound emanate from thehead 160 near the web pad 310 displaying the videoconferencing.

The interfacing module 300 may include a microphone 330 (or micro-phonearray) for receiving sound inputs and one or more speakers 340 disposedon the robot body 110 for delivering sound outputs.

Referring to FIGS. 1-3C, to achieve reliable and robust autonomousmovement, the sensor system 400 may include several different types ofsensors which can be used in conjunction with one another to create aperception of the robot's environment sufficient to allow the robot 100to make intelligent decisions about actions to take in that environment.The sensor system 400 may include one or more types of sensors supportedby the robot body 110, which may include obstacle detection obstacleavoidance (ODOA) sensors, communication sensors, navigation sensors,etc. For example, these sensors may include, but are not limited to,proximity sensors, contact sensors, three-dimensional imaging/depth mapsensors, a camera (e.g., visible light and/or infrared camera), sonar,radar, Light Detection And Ranging (LIDAR), which can entail opticalremote sensing that measures properties of scattered light to find rangeand/or other information of a distant target), Laser Detection andRanging (LADAR), etc. In some implementations, the sensor system 400includes ranging sonar sensors 410 (e.g., nine about a perimeter of thebase 120), proximity cliff detectors 420, contact sensors 430 (FIG. 4A),a laser scanner 440, one or more three-dimensional imaging/depth sensors450, and an imaging sonar 460.

In some implementations, the sensor system 400 includes a set or anarray of proximity sensors 410, 420 in communication with the controller500 and arranged in one or more zones or portions of the robot 100(e.g., disposed on or near the base body portion 124 a, 124 b, 124 c ofthe robot body 110) for detecting any nearby or intruding obstacles. Theproximity sensors 410, 420 may be converging infrared (IR)emitter-sensor elements, sonar sensors, ultrasonic sensors, and/orimaging sensors (e.g., 3D depth map image sensors) that provide a signalto the controller 500 when an object is within a given range of therobot 100.

In the example shown in FIGS. 4A-4C, the robot 100 includes an array ofsonar-type proximity sensors 410 disposed (e.g., substantiallyequidistant) around the base 124 of body 120 and arranged with an upwardfield of view. First, second, and third sonar proximity sensors 410 a,410 b, 410 c are disposed on or near the first (forward) base bodyportion 124 a, with at least one of the sonar proximity sensors near aradially outermost edge 125 a of the first base 124 a of body 120.Fourth, fifth, and sixth sonar proximity sensors 410 d, 410 e, 410 f aredisposed on or near the second (right) base body portion 124 b, with atleast one of the sonar proximity sensors near a radially outermost edge125 b of the second base 124 b of body 120. Seventh, eighth, and ninthsonar proximity sensors 410 g, 410 h, 410 i are disposed on or near thethird (left) base body portion 124 c, with at least one of the sonarproximity sensors near a radially outermost edge 125 c of the third base124 c of body 120. This configuration provides at least three zones ofdetection.

In some examples, the set of sonar proximity sensors 410 (e.g., 410a-410 i) disposed around the base 124 of body 120 are arranged to pointupward (e.g., substantially in the Z direction) and optionally angledoutward away from the Z axis, thus creating a detection curtain 412around the robot 100. Each sonar proximity sensor 410 a-410 i may have ashroud or emission guide 414 that guides the sonar emission upward or atleast not toward the other portions of the robot body 110 (e.g., so asnot to detect movement of the robot body 110 with respect to itself).The emission guide 414 may define a shell or half-shell shape. In theexample shown, the base 124 of body 120 extends laterally beyond the leg130, and the sonar proximity sensors 410 (e.g., 410 a-410 i) aredisposed on the base 124 of body 120 (e.g., substantially along aperimeter of the base 124 of body 120) around the leg 130. Moreover, theupward pointing sonar proximity sensors 410 are spaced to create acontinuous or substantially continuous sonar detection curtain 412around the leg 130. The sonar detection curtain 412 can be used todetect obstacles having elevated lateral protruding portions, such astable tops, shelves, etc.

The upward looking sonar proximity sensors 410 provide the ability tosee objects that are primarily in the horizontal plane, such as tabletops. These objects, due to their aspect ratio, may be missed by othersensors of the sensor system, such as the laser scanner 440 or imagingsensors 450, and as such, can pose a problem to the robot 100. Theupward viewing sonar proximity sensors 410 arranged around the perimeterof the base 120 provide a means for seeing or detecting those type ofobjects/obstacles. Moreover, the sonar proximity sensors 410 can beplaced around the widest points of the base perimeter and angledslightly outwards, so as not to be occluded or obstructed by the torso140 or head 160 of the robot 100, thus not resulting in false positivesfor sensing portions of the robot 100 itself. In some implementations,the sonar proximity sensors 410 are arranged (upward and outward) toleave a volume about the torso 140 outside of a field of view of thesonar proximity sensors 410 and thus free to receive mounted payloads oraccessories, such as the basket 360. The sonar proximity sensors 410 canbe recessed into the base body 124 to provide visual concealment and noexternal features to snag on or hit obstacles.

The sensor system 400 may include one or more sonar proximity sensors410 (e.g., a rear proximity sensor 410 j) directed rearward (e.g.,opposite to the forward drive direction F) for detecting obstacles whilebacking up. The rear sonar proximity sensor 410 j may include anemission guide 414 to direct its sonar detection field 412. Moreover,the rear sonar proximity sensor 410 j can be used for ranging todetermine a distance between the robot 100 and a detected object in thefield of view of the rear sonar proximity sensor 410 j (e.g., as“back-up alert”). In some examples, the rear sonar proximity sensor 410j is mounted recessed within the base 124 of body 120 so as not toprovide any visual or functional irregularity in the housing form.

Referring to FIGS. 2 and 4B, in some implementations, the robot 100includes cliff proximity sensors 420 arranged near or about the drivewheels 210 a, 210 b, 210 c, so as to allow cliff detection before thedrive wheels 210 a, 210 b, 210 c encounter a cliff (e.g., stairs). Forexample, a cliff proximity sensors 420 can be located at or near each ofthe radially outermost edges 125 a-c of the base bodies 124 a-c and inlocations therebetween. In some cases, cliff sensing is implementedusing infrared (IR) proximity or actual range sensing, using an infraredemitter 422 and an infrared detector 424 angled toward each other so asto have overlapping emission and detection fields, and hence a detectionzone, at a location where a floor should be expected. IR proximitysensing can have a relatively narrow field of view, may depend onsurface albedo for reliability, and can have varying range accuracy fromsurface to surface. As a result, multiple discrete sensors can be placedabout the perimeter of the robot 100 to adequately detect cliffs frommultiple points on the robot 100. Moreover, IR proximity based sensorstypically cannot discriminate between a cliff and a safe event, such asjust after the robot 100 climbs a threshold.

The cliff proximity sensors 420 can detect when the robot 100 hasencountered a falling edge of the floor, such as when it encounters aset of stairs. The controller 500 (executing a control system) mayexecute behaviors that cause the robot 100 to take an action, such aschanging its direction of travel, when an edge is detected. In someimplementations, the sensor system 400 includes one or more secondarycliff sensors (e.g., other sensors configured for cliff sensing andoptionally other types of sensing). The cliff detecting proximitysensors 420 can be arranged to provide early detection of cliffs,provide data for discriminating between actual cliffs and safe events(such as climbing over thresholds), and be positioned down and out sothat their field of view includes at least part of the robot body 110and an area away from the robot body 110. In some implementations, thecontroller 500 executes cliff detection routine that identifies anddetects an edge of the supporting work surface (e.g., floor), anincrease in distance past the edge of the work surface, and/or anincrease in distance between the robot body 110 and the work surface.This implementation allows: 1) early detection of potential cliffs(which may allow faster mobility speeds in unknown environments); 2)increased reliability of autonomous mobility since the controller 500receives cliff imaging information from the cliff detecting proximitysensors 420 to know if a cliff event is truly unsafe or if it can besafely traversed (e.g., such as climbing up and over a threshold); 3) areduction in false positives of cliffs (e.g., due to the use of edgedetection versus the multiple discrete IR proximity sensors with anarrow field of view). Additional sensors arranged as “wheel drop”sensors can be used for redundancy and for detecting situations where arange-sensing camera cannot reliably detect a certain type of cliff.

Threshold and step detection allows the robot 100 to effectively planfor either traversing a climbable threshold or avoiding a step that istoo tall. This can be the same for random objects on the work surfacethat the robot 100 may or may not be able to safely traverse. For thoseobstacles or thresholds that the robot 100 determines it can climb,knowing their heights allows the robot 100 to slow down appropriately,if deemed needed, to allow for a smooth transition in order to maximizesmoothness and minimize any instability due to sudden accelerations. Insome implementations, threshold and step detection is based on objectheight above the work surface along with geometry recognition (e.g.,discerning between a threshold or an electrical cable versus a blob,such as a sock). Thresholds may be recognized by edge detection. Thecontroller 500 may receive imaging data from the cliff detectingproximity sensors 420 (or another imaging sensor on the robot 100),execute an edge detection routine, and issue a drive command based onresults of the edge detection routine. The controller 500 may usepattern recognition to identify objects as well. Threshold detectionallows the robot 100 to change its orientation with respect to thethreshold to maximize smooth step climbing ability.

The proximity sensors 410, 420 may function alone, or as an alternative,may function in combination with one or more contact sensors 430 (e.g.,bump switches) for redundancy. For example, one or more contact or bumpsensors 430 on the robot body 110 can detect if the robot 100 physicallyencounters an obstacle. Such sensors may use a physical property such ascapacitance or physical displacement within the robot 100 to determinewhen it has encountered an obstacle. In some implementations, each basebody portion 124 a, 124 b, 124 c of the base 120 has an associatedcontact sensor 430 (e.g., capacitive sensor, read switch, etc.) thatdetects movement of the corresponding base body portion 124 a, 124 b,124 c with respect to the base chassis 122 (see e.g., FIG. 4A). Forexample, each base 124 a of body 120-c may move radially with respect tothe Z axis of the base chassis 122, so as to provide three-way bumpdetection.

Referring again to FIGS. 1-4C, in some implementations, the sensorsystem 400 includes a laser scanner 440 mounted on a forward portion ofthe robot body 110 and in communication with the controller 500. In theexamples shown, the laser scanner 440 is mounted on the base 124 of body120 facing forward (e.g., having a field of view along the forward drivedirection F) on or above the first base 124 a of body 120 (e.g., to havemaximum imaging coverage along the drive direction F of the robot).Moreover, the placement of the laser scanner on or near the front tip ofthe triangular base 120 means that the external angle of the roboticbase (e.g., 300°) is greater than a field of view 442 of the laserscanner 440 (e.g., ˜285°), thus preventing the base 120 from occludingor obstructing the detection field of view 442 of the laser scanner 440.The laser scanner 440 can be mounted recessed within the base body 124as much as possible without occluding its fields of view, to minimizeany portion of the laser scanner sticking out past the base body 124(e.g., for aesthetics and to minimize snagging on obstacles).

The laser scanner 440 scans an area about the robot 100 and thecontroller 500, using signals received from the laser scanner 440, andcreates an environment map or object map of the scanned area. Thecontroller 500 may use the object map for navigation, obstacledetection, and obstacle avoidance. Moreover, the controller 500 may usesensory inputs from other sensors of the sensor system 400 for creatingan object map and/or for navigation.

In some examples, the laser scanner 440 is a scanning LIDAR, which mayuse a laser that quickly scans an area in one dimension, as a “main”scan line, and a time-of-flight imaging element that uses a phasedifference or similar technique to assign a depth to each pixelgenerated in the line (returning a two-dimensional depth line in theplane of scanning). In order to generate a three-dimensional map, theLIDAR can perform an “auxiliary” scan in a second direction (forexample, by “nodding” the scanner). This mechanical scanning techniquecan be complemented, if not supplemented, by technologies such as the“Flash” LIDAR/LADAR and “Swiss Ranger” type focal plane imaging elementsensors, techniques which use semiconductor stacks to permit time offlight calculations for a full two-dimensional matrix of pixels toprovide a depth at each pixel, or even a series of depths at each pixel(with an encoded illuminator or illuminating laser).

The sensor system 400 may include one or more three-dimensional imagesensors 450 in communication with the controller 500. If thethree-dimensional image sensor 450 has a limited field of view, thecontroller 500 or the sensor system 400 can actuate thethree-dimensional image sensor 450 a in a side-to-side scanning mannerto create a relatively wider field of view to perform robust obstacledetection/obstacle avoidance (ODOA). Referring to FIGS. 1-3B, in someimplementations, the robot 100 includes a scanning three-dimensionalimage sensor 450 a mounted on a forward portion of the robot body 110with a field of view along the forward drive direction F (e.g., to havemaximum imaging coverage along the drive direction F of the robot). Thescanning three-dimensional image sensor 450 a can be used primarily forODOA. In the example shown, the scanning three-dimensional image sensor450 a is mounted on the torso 140 underneath the shoulder 142 or on thebottom surface 144 and recessed within the torso 140 (e.g., flush orpast the bottom surface 144), as shown in FIG. 2, for example, toprevent user contact with the scanning three-dimensional image sensor450 a. The scanning three-dimensional image sensor 450 can be arrangedto aim substantially downward and away from the robot body 110, so as tohave a downward field of view 452 in front of the robot 100 for ODOA(e.g., with obstruction by the base 120 or other portions of the robotbody 110). Placement of the scanning three-dimensional image sensor 450a on or near a forward edge of the torso 140 allows the field of view ofthe three-dimensional image sensor 450 (e.g., ˜285°) to be less than anexternal surface angle of the torso 140 (e.g., 300°) with respect to thethree-dimensional image sensor 450, thus preventing the torso 140 fromoccluding or obstructing the detection field of view 452 of the scanningthree-dimensional image sensor 450 a. Moreover, the scanningthree-dimensional image sensor 450 a (and associated actuator) can bemounted recessed within the torso 140 as much as possible withoutoccluding its fields of view (e.g., also for aesthetics and to minimizesnagging on obstacles). The distracting scanning motion of the scanningthree-dimensional image sensor 450 a is not visible to a user, creatinga less distracting interaction experience. Unlike a protruding sensor orfeature, the recessed scanning three-dimensional image sensor 450 a willnot tend to have unintended interactions with the environment (snaggingon people, obstacles, etc.), especially when moving or scanning, asvirtually no moving part extends beyond the envelope of the torso 140.

In some implementations, the sensor system 400 includes additionalthree-dimensional image sensors 450 disposed on the base 124 of body120, the leg 130, the neck 150, and/or the head 160. In the exampleshown in FIG. 1, the robot 100 includes three-dimensional image sensors450 on the base 124 of body 120, the torso 140, and the head 160. In theexample shown in FIG. 3A, the robot 100 includes three-dimensional imagesensors 450 on the base 124 of body 120, the torso 140, and the head160. In the example shown in FIG. 3B, the robot 100 includesthree-dimensional image sensors 450 on the leg 130, the torso 140, andthe neck 150. Other configurations are possible as well. Onethree-dimensional image sensor 450 (e.g., on the neck 150 and over thehead 160) can be used for people recognition, gesture recognition,and/or videoconferencing, while another three-dimensional image sensor450 (e.g., on the base 120 and/or the leg 130) can be used fornavigation and/or obstacle detection and obstacle avoidance.

A forward facing three-dimensional image sensor 450 disposed on the neck150 and/or the head 160 can be used for person, face, and/or gesturerecognition of people about the robot 100. For example, using signalinputs from the three-dimensional image sensor 450 on the head 160, thecontroller 500 may recognize a user by creating a three-dimensional mapof the viewed/captured user's face and comparing the createdthree-dimensional map with known three-dimensional images of people'sfaces and determining a match with one of the known three-dimensionalfacial images. Facial recognition may be used for validating users asallowable users of the robot 100. Moreover, one or more of thethree-dimensional image sensors 450 can be used for determining gesturesof a person viewed by the robot 100, and optionally reacting based onthe determined gesture(s) (e.g., hand pointing, waving, and/or handsignals). For example, the controller 500 may issue a drive command inresponse to a recognized hand pointing in a particular direction.

The three-dimensional image sensors 450 may be capable of producing thefollowing types of data: (i) a depth map, (ii) a reflectivity basedintensity image, and/or (iii) a regular intensity image. Thethree-dimensional image sensors 450 may obtain such data by imagepattern matching, measuring the flight time and/or phase delay shift forlight emitted from a source and reflected off of a target.

In some implementations, reasoning or control software, executable on aprocessor (e.g., of the robot controller 500), uses a combination ofalgorithms executed using various data types generated by the sensorsystem 400. The reasoning software processes the data collected from thesensor system 400 and outputs data for making navigational decisions onwhere the robot 100 can move without colliding with an obstacle, forexample. By accumulating imaging data over time of the robot'ssurroundings, the reasoning software can in turn apply effective methodsto selected segments of the sensed image(s) to improve depthmeasurements of the three-dimensional image sensors 450. This mayinclude using appropriate temporal and spatial averaging techniques.

The reliability of executing robot collision-free moves may be based on:(i) a confidence level built by high-level reasoning over time and (ii)a depth-perceptive sensor that accumulates three major types of data foranalysis: (a) a depth image, (b) an active illumination image and (c) anambient illumination image. Algorithms cognizant of the different typesof data can be executed on each of the images obtained by thedepth-perceptive image sensor 450. The aggregate data may improve theconfidence level compared to a system using only one of the kinds ofdata.

The three-dimensional image sensors 450 may obtain images containingdepth and brightness data from a scene about the robot 100 (e.g., asensor view portion of a room or work area) that contains one or moreobjects. The controller 500 may be configured to determine occupancydata for the object based on the captured reflected light from thescene. Moreover, the controller 500, in some examples, issues a drivecommand to the drive system 200 based at least in part on the occupancydata to circumnavigate obstacles (i.e., the object in the scene). Thethree-dimensional image sensors 450 may repeatedly capture scene depthimages for real-time decision-making by the controller 500 to navigatethe robot 100 about the scene without colliding into any objects in thescene. For example, the speed or frequency in which the depth image datais obtained by the three-dimensional image sensors 450 may be controlledby a shutter speed of the three-dimensional image sensors 450. Inaddition, the controller 500 may receive an event trigger (e.g., fromanother sensor component of the sensor system 400, such as proximitysensor 410, 420, notifying the controller 500 of a nearby object orhazard. The controller 500, in response to the event trigger, can causethe three-dimensional image sensors 450 to increase a frequency at whichdepth images are captured and occupancy information is obtained.

In some implementations, the robot includes a sonar scanner 460 foracoustic imaging of an area surrounding the robot 100. In the examplesshown in FIGS. 1 and 2, the sonar scanner 460 is disposed on a forwardportion of the base 124 of body 120.

Referring to FIGS. 1-3B, in some implementations, the robot 100 uses thelaser scanner or laser range finder 440 for redundant sensing, as wellas a rear-facing sonar proximity sensor 410 j for safety, both of whichare oriented parallel to the ground G. The robot 100 may include firstand second three-dimensional image sensors 450 a, 450 b (depth cameras)to provide robust sensing of the environment around the robot 100. Thefirst three-dimensional image sensor 450 a is mounted on the torso 140and pointed downward at a fixed angle to the ground G. By angling thefirst three-dimensional image sensor 450 a downward, the robot 100receives dense sensor coverage in an area immediately forward oradjacent to the robot 100, which is relevant for short-term travel ofthe robot 100 in the forward direction. The rear-facing sonar 410 jprovides object detection when the robot travels backward. If backwardtravel is typical for the robot 100, the robot 100 may include a third3D image sensor 450 facing downward and backward to provide dense sensorcoverage in an area immediately rearward or adjacent to the robot 100.

The second three-dimensional image sensor 450 b is mounted on the head160, which can pan and tilt via the neck 150. The secondthree-dimensional image sensor 450 b can be useful for remote drivingsince it allows a human operator to see where the robot 100 is going.The neck 150 enables the operator tilt and/or pan the secondthree-dimensional image sensor 450 b to see both close and distantobjects. Panning the second three-dimensional image sensor 450 bincreases an associated horizontal field of view. During fast travel,the robot 100 may tilt the second three-dimensional image sensor 450 bdownward slightly to increase a total or combined field of view of boththree-dimensional image sensors 450 a, 450 b, and to give sufficienttime for the robot 100 to avoid an obstacle (since higher speedsgenerally mean less time to react to obstacles). At slower speeds, therobot 100 may tilt the second three-dimensional image sensor 450 bupward or substantially parallel to the ground G to track a person thatthe robot 100 is meant to follow. Moreover, while driving at relativelylow speeds, the robot 100 can pan the second three-dimensional imagesensor 450 b to increase its field of view around the robot 100. Thefirst three-dimensional image sensor 450 a can stay fixed (e.g., notmoved with respect to the base 120) when the robot is driving to expandits perceptual range. Additionally and/or alternatively, the firstthree-dimensional image sensor 450 a can scan at low speeds in order todetect potential obstacles around the robot when it is maneuvering. Insome examples, the height of the first three-dimensional image sensor450 a can be adjusted upward, such as through the use of a Z-lift, inorder to optimize the field of view of the first three-dimensionalsensor 450 a.

In some implementations, at least one of three-dimensional image sensors450 can be a volumetric point cloud imaging device (such as a speckle ortime-of-flight camera) positioned on the robot 100 at a height ofgreater than one or two feet above the ground (or at a height of aboutone or two feet above the ground) and directed to obtain a point cloudfrom a volume of space including a floor plane in a direction ofmovement of the robot (via the omni-directional drive system 200). Inthe examples shown in FIGS. 1 and 3, the first three-dimensional imagesensor 450 a can be positioned on the base 120 at height of greater thanone or two feet above the ground and aimed along the forward drivedirection F to capture images (e.g., volumetric point cloud) of a volumeincluding the floor while driving (e.g., for obstacle detection andobstacle avoidance). The second three-dimensional image sensor 450 b isshown mounted on the head 160 (e.g., at a height greater than aboutthree or four feet above the ground), so as to obtain skeletalrecognition and definition point clouds from a volume of space adjacentthe robot 100. The controller 500 may execute skeletal/digitalrecognition software to analyze data of the captured volumetric pointclouds.

Referring to FIGS. 3A-4C, the sensor system 400 may include an inertialmeasurement unit (IMU) 470 in communication with the controller 500 tomeasure and monitor a moment of inertia of the robot 100 with respect tothe overall center of gravity CG_(R) of the robot 100.

The controller 500 may monitor any deviation in feedback from the IMU470 from a threshold signal corresponding to normal unencumberedoperation. For example, if the robot begins to pitch away from anupright position, it may be “clotheslined” or otherwise impeded, orsomeone may have suddenly added a heavy payload. In these instances, itmay be necessary to take urgent action (including, but not limited to,evasive maneuvers, recalibration, and/or issuing an audio/visualwarning) in order to assure safe operation of the robot 100.

Since robot 100 may operate in a human environment, it may interact withhumans and operate in spaces designed for humans (and without regard forrobot constraints). The robot 100 can limit its drive speeds andaccelerations when in a congested, constrained, or highly dynamicenvironment, such as at a cocktail party or busy hospital. However, therobot 100 may encounter situations where it is safe to drive relativelyfast, as in a long empty corridor, but yet be able to deceleratesuddenly, as when something crosses the robot's motion path.

When accelerating from a stop, the controller 500 may take into accounta moment of inertia of the robot 100 from its overall center of gravityCG_(R) to prevent robot tipping. The controller 500 may use a model ofits pose, including its current moment of inertia. When payloads aresupported, the controller 500 may measure a load impact on the overallcenter of gravity CG_(R) and monitor movement of the robot moment ofinertia. For example, the torso 140 and/or neck 150 may include straingauges to measure strain. If this is not possible, the controller 500may apply a test torque command to the drive wheels 210 and measureactual linear and angular acceleration of the robot using the IMU 470,in order to experimentally determine safe limits.

During a sudden deceleration, a commanded load on the second and thirddrive wheels 210 b, 210 c (the rear wheels) is reduced, while the firstdrive wheel 210 a (the front wheel) slips in the forward drive directionand supports the robot 100. If the loading of the second and third drivewheels 210 b, 210 c (the rear wheels) is asymmetrical, the robot 100 may“yaw” which will reduce dynamic stability. The IMU 470 (e.g., a gyro)can be used to detect this yaw and command the second and third drivewheels 210 b, 210 c to reorient the robot 100.

Referring to FIG. 5, in some implementations, the controller 500executes a control system 510, which includes a behavior system 510 aand a control arbitration system 510 b in communication with each other.The control arbitration system 510 b allows applications 520 to bedynamically added and removed from the control system 510, andfacilitates allowing applications 520 to each control the robot 100without needing to know about any other applications 520. In otherwords, the control arbitration system 510 b provides a simpleprioritized control mechanism between applications 520 and resources 530of the robot 100. The resources 530 may include the drive system 200,the sensor system 400, and/or any payloads or controllable devices incommunication with the controller 500.

The applications 520 can be stored in memory of or communicated to therobot 100, to run concurrently on (e.g., a processor) and simultaneouslycontrol the robot 100. The applications 520 may access behaviors 512 ofthe behavior system 510 a. The independently deployed applications 520are combined dynamically at runtime and share robot resources 530 (e.g.,drive system 200, arm(s), head(s), etc.) of the robot 100. A low-levelpolicy is implemented for dynamically sharing the robot resources 530among the applications 520 at run-time. The policy determines whichapplication 520 has control of the robot resources 530 required by thatapplication 520 (e.g. it creates a priority hierarchy among theapplications 520). Applications 520 can start and stop dynamically andrun completely independently of each other. The control system 510 alsoallows for complex behaviors 512 which can be combined together toassist each other.

The control arbitration system 510 b includes one or more resourcecontrollers 540, a robot manager 550, and one or more control arbiters560. These components do not need to be in a common process or computer,and do not need to be started in any particular order. The resourcecontroller 540 component provides an interface to the controlarbitration system 510 b for applications 520. There is an instance ofthis component for every application 520. The resource controller 540abstracts and encapsulates away the complexities of authentication,distributed resource control arbiters, command buffering, and the like.The robot manager 550 coordinates the prioritization of applications520, by controlling which application 520 has exclusive control of anyof the robot resources 530 at any particular time. Since this is thecentral coordinator of information, there is only one instance of therobot manager 550 per robot. The robot manager 550 implements a prioritypolicy, which has a linear prioritized order of the resource controllers540, and keeps track of the resource control arbiters 560 that providehardware control. The control arbiter 560 receives the commands fromevery application 520 generates a single command based on theapplications' priorities and publishes it for its associated resources530. The control arbiter 560 also receives state feedback from itsassociated resources 530 and sends it back up to the applications 520.The robot resources 530 may be a network of functional modules (e.g.actuators, drive systems, and groups thereof) with one or more hardwarecontrollers. The commands of the control arbiter 560 are specific to theresource 530 to carry out specific actions.

A dynamics model 570 executable on the controller 500 can be configuredto compute the center of gravity (CG), moments of inertia, and crossproducts of inertia of various portions of the robot 100 for theassessing a current robot state. The dynamics model 570 may also modelthe shapes, weight, and/or moments of inertia of these components. Insome examples, the dynamics model 570 communicates with the IMU 470 orportions of one (e.g., accelerometers and/or gyros) disposed on therobot 100 and in communication with the controller 500 for calculatingthe various centers of gravity of the robot 100. The dynamics model 570can be used by the controller 500, along with other programs 520 orbehaviors 512 to determine operating envelopes of the robot 100 and itscomponents.

Each application 520 has an action selection engine 580 and a resourcecontroller 540, one or more behaviors 512 connected to the actionselection engine 580, and one or more action models 590 connected toaction selection engine 580. The behavior system 510 a providespredictive modeling and allows the behaviors 512 to collaborativelydecide on the robot's actions by evaluating possible outcomes of robotactions. In some examples, a behavior 512 is a plug-in component thatprovides a hierarchical, state-full evaluation function that couplessensory feedback from multiple sources with a-priori limits andinformation into evaluation feedback on the allowable actions of therobot. Since the behaviors 512 can be plugged into the application 520(e.g., residing inside or outside of the application 520), they can beremoved and added without having to modify the application 520 or anyother part of the control system 510. Each behavior 512 is a standalonepolicy. To make behaviors 512 more powerful, it is possible to attachthe output of multiple behaviors 512 together into the input of anotherso as to have complex combination functions. The behaviors 512 areintended to implement manageable portions of the total cognizance of therobot 100.

The action selection engine 580 is the coordinating element of thecontrol system 510 and runs a fast, optimized action selection cycle(prediction/correction cycle) searching for the best action given theinputs of all the behaviors 512. The action selection engine 580 hasthree phases: nomination, action selection search, and completion. Inthe nomination phase, each behavior 512 is notified that the actionselection cycle has started and is provided with the cycle start time,the current state, and limits of the robot actuator space. Based oninternal policy or external input, each behavior 512 decides whether ornot it wants to participate in this action selection cycle. During thisphase, a list of active behavior primitives is generated whose inputwill affect the selection of the commands to be executed on the robot100.

In the action selection search phase, the action selection engine 580generates feasible outcomes from the space of available actions, alsoreferred to as the action space. The action selection engine 580 usesthe action models 590 to provide a pool of feasible commands (withinlimits) and corresponding outcomes as a result of simulating the actionof each command at different time steps with a time horizon in thefuture. The action selection engine 580 calculates a preferred outcome,based on the outcome evaluations of the behaviors 512, and sends thecorresponding command to the control arbitration system 510 b andnotifies the action model 590 of the chosen command as feedback.

In the completion phase, the commands that correspond to a collaborativebest scored outcome are combined together as an overall command, whichis presented to the resource controller 540 for execution on the robotresources 530. The best outcome is provided as feedback to the activebehaviors 512, to be used in future evaluation cycles.

Received sensor signals from the sensor system 400 can causeinteractions with one or more behaviors 512 to execute actions. Forexample, using the control system 510, the controller 500 selects anaction (or move command) for each robotic component (e.g., motor oractuator) from a corresponding action space (e.g., a collection ofpossible actions or moves for that particular component) to effectuate acoordinated move of each robotic component in an efficient manner thatavoids collisions with itself and any objects about the robot 100, whichthe robot 100 is aware of. The controller 500 can issue a coordinatedcommand over robot network, such as an EtherIO network, as described inU.S. Ser. No. 61/305,069, filed Feb. 16, 2010, the entire contents ofwhich are hereby incorporated by reference.

FIG. 6A provides a schematic view of an exemplary robot system 600having one or more telepresence robots 100 in communication with abridge 602, which communicates with a local robot endpoint server 604 aand a remote endpoint server 604 b (e.g., such as the cloud computingservice 720 (FIG. 7)). The local robot endpoint server 604 acommunicates with a local technician computing device 606 and the remoteendpoint server 604 b communicates with a remote operator computingdevice 608.

Referring to FIGS. 2 and 4C, in some implementations, the robot 100includes multiple antennas. In the examples shown, the robot 100includes a first antenna 490 a and a second antenna 490 b both disposedon the base 120 (although the antennas may be disposed at any other partof the robot 100, such as the leg 130, the torso 140, the neck 150,and/or the head 160). The use of multiple antennas provides robustsignal reception and transmission. The use of multiple antennas providesthe robot 100 with multiple-input and multiple-output (MIMO) which isthe use of multiple antennas for a transmitter and/or a receiver toimprove communication performance. MIMO offers significant increases indata throughput and link range without additional bandwidth or transmitpower. It achieves this by higher spectral efficiency (more bits persecond per hertz of bandwidth) and link reliability or diversity(reduced fading). Because of these properties, MIMO is an important partof modern wireless communication standards such as IEEE 802.11n (Wifi),4G, 3GPP Long Term Evolution, WiMAX and HSPA+. Moreover, the robot 100can act as a Wi-Fi bridge, hub or hotspot for other electronic devicesnearby. The mobility and use of MIMO of the robot 100 can allow therobot to serve as a relatively reliable Wi-Fi bridge 602.

Referring to FIGS. 6A and 6B, a teleoperation software application 601executes on at least one of the robot controller 500, the local robotendpoint server 604 a, the remote endpoint server 604 b, the localtechnician computing device 606 and the remote operator computing device608. In some examples, a portion of the teleoperation softwareapplication 601 executes on one or more of the aforementioned devices.The teleoperation software application 601 allows one or more users tointeract with the robot 100 (e.g., to drive the robot 100) and/orremotely with other people or objects adjacent the robot 100 throughtelepresence features of the robot 100.

FIG. 6C provides a schematic view of an exemplary user interface 605 ofthe teleoperation software application 601 that can be rendered on adisplay, such as the touch screen 312 of the web pad 310 and/or theremote operator computing device 608, for controlling navigation,telepresence, and/or other aspects of the robot 100. The user interface605 includes a remote video feed window 610 displaying a remote view612, such as a video feed of a patient 614. The video feed may begenerated by one of the cameras 320, 450 on the robot 100. The userinterface 605 may display a plan view map window 620 having a map 622 ofthe local area in which the robot 100 is operating. In the exampleshown, the map 622 displayed in the plan view map window 620 is atwo-dimensional, top-down map 622 a (FIG. 6D); however, other types ofmaps are possible as well. The user interface 605 may also include alocal video window 630 displaying a local view 632, such as a video feedof the user (e.g., remote from the robot 100). The video feed displayedin the local video window 630 may be transmitted to the robot 100 anddisplayed to the patient 614 using a display device, such as the web pad310 on the robot 100.

A dashboard 640 may provide information regarding the orientation of therobot 100, an indication of the robot's battery charge, an indication ofthe strength of a wireless data signal, and/or an indication of thenetwork quality. The orientation of the robot 100 may be indicated by anicon 642 displaying the orientation of the head 160 of the robot 100with respect to the torso 140 or the base 120. Such an indication mayassist a user in orienting the robot 100 to view items of interest. Therange of motion of the robot head 160 may be limited. Accordingly,certain implementations may display an indication of the rotationalposition of the head 160 and a range of motion of the head 160.

Media controls 647 may allow the user to interact with the patient 614using various types of media and to acquire and store media documentingthe interactions of the user and the patient 614. The media controls 647may allow the user to play audio and/or video clips, for example, thatmay be used to educate the patient 614 about a medical condition orprocedure. Still photographs may be acquired using a camera 320, 450 ofthe robot 100 in order to document various conditions. Further, therobot 100 may acquire audio (e.g., using the microphone 330) or video(e.g., using the camera 320) documenting the user's interaction with thepatient 614 and optionally storing the acquired audio/video in memory ofthe controller 500 and/or transmitting the acquired audio/video to aremote device or cloud service.

In some implementations, the media controls 647 allow the user to managetemporary connectivity issues. For example, upon the unexpecteddisconnection of a session, video recording may begin. The robot 100 maycontinue recording video and saving it to local memory, such as that ofthe controller 500. Upon an unexpected disconnection, a message may bedisplayed by the robot, such as “Session terminated-Video recordingcontinuing . . . .” A button below may be displayed with a caption of“Stop recording.” A nurse on the robot side may touch the “Stoprecording” button (e.g., on the touch screen 312) and terminate thelocal recording. Otherwise, the recording may continue for a specifiedtime interval. If the same user logs back into the robot 100 in thespecified time interval, the record button on the remote station mayshow that recording is in progress. When the robot's local recording iscomplete, it may begin to transmit the video file to a remote station orother location that may be accessible to the disconnected user.Accordingly, the user may be able to see what events transpired duringthe time that the session was interrupted.

In the example shown in FIG. 6C, the remote video feed window 610occupies a relatively large portion of the display area. The userinterface 605 may have the remote video feed window 610 rendered at a640×480 pixel resolution, the local video window 630 at a 320×240 pixelresolution, and the plan view map window 620 at a 530×200 pixelresolution. Accordingly, this view may be most appropriate when the useris communicating with the patient 614 and/or manually driving the robot100. The layout of the default user interface 605 a shown in FIG. 6C mayallow the user to swap the contents of the plan view map window 620 withthe remote video feed window 610. The view may be swapped, for example,by double-clicking on the map window 620. The windows could later beswapped back by double-clicking on the remote video feed window 610.

Alternative screen layouts may be displayed to a user as may beappropriate for the task being performed by the user. In the exampleshown in FIG. 6D, in anticipation of directing the robot 100 to moveusing semi-autonomous navigation from one location to another, the sizeof the plan view map window 620 is increased. For example, the userinterface 605, 605 a shown in FIG. 6C can be a default state for patientinteraction and the user interface 605, 605 b shown in FIG. 6D can be analternate state for robot navigation.

A map view switch button 645 of the user interface may allow the user toinvoke the alternative user interface 605 b that includes a relativelylarger map window 620. For example, the user interface 605 b shown inFIG. 6D may be primarily used to manually drive the robot 100 or toautonomously navigate to a desired destination Clicking the map viewswitch button 645 again takes the user back to the default userinterface 605 a, which may be used when actively performing a medicalconsultation. Accordingly, a user may emphasize or de-emphasize (e.g.,maximize or minimize) the plan view map window 620 as desired. Certainof the windows shown in the alternate user interface 605 b are alsodisplayed, such as the remote video feed window 610 and the local videowindow 630. In some examples, the plan view map window 620 may bedisplayed at a 880×700 pixel resolution, the remote video feed window610 may be displayed at a 320×240 pixel resolution, and the local videowindow 630 may be displayed at a 160×120 pixel resolution.

Referring to FIG. 6D, the plan view map window 620 may provide a robotlocation icon 650 designating a location of the robot 100 within thelocal environment. A user may click on or touch a point on the displayedmap 622 in order to cause the robot to semi-autonomously or autonomouslynavigate to the selected point. In some examples, the user may zoom inand out using a mouse wheel while the cursor is over the plan view mapwindow 620 or by touch gestures, when displayed on a touch screen 312.

Simultaneous localization and mapping (SLAM) technology may utilizelaser range scanners, odometry, acoustic range finders, or all of theabove to build a map of the local environment and place the robot 100 onthe map. Images recorded by a robot 100 (e.g., via the camera 320 orthree-dimensional image sensor 450) as it traverses an environment maybe stored in an internal database (e.g., of the controller 500) and/or aremote database (e.g., a cloud service). When the robot 100 reacquiresan image currently in the database, the algorithm resets the robot'scurrent position to that which was recorded when the landmark wasoriginally entered in the database. This method helps to counter theinherent drift of wheel encoder odometry. Systems may also utilize RFIDchips and/or triangulation of wireless access points. Further, the namesor identifying numbers of specific rooms may be associated withlocations on the map. Image data can accumulate over time and, as a costsavings or space savings, the robot 100 may use remote data storageand/or remote processing for storing and/or processing the image data,respectively. For example, an RFID reader may detect RFID chipsassociated with coordinates on a plan view map in order to identify acurrent location of a robot. An “RFID chip” may include an RFID deviceor an RFID “tag” as is understood by those having skill in the art. TheRFID chips may be embodied as passive, active, or battery assistedpassive (BAP) RFID chips.

FIG. 7 provides a schematic view of an exemplary robot systemarchitecture 700, which may include the robot 100 (or a portion thereof,such as the controller 500 or drive system 200), a computing device 310(e.g., detachably or fixedly attached to the head 160), a cloud 720(i.e., cloud computing service), and a portal 730. The computing device310 may execute one or more robot applications 710, which may includesoftware applications (e.g., stored in memory and executable on aprocessor) for security, medicine compliance, telepresence, behavioralcoaching, social networking, active alarm, home management, etc. Thecomputing device 310 may provide communication capabilities (e.g.,secure wireless connectivity and/or cellular communication), refinedapplication development tools, speech recognition, and person or objectrecognition capabilities. The computing device 310 in some examplesutilizes an interaction/COMS featured operating system, such as Androidprovided by Google Inc., iOS provided by Apple, Inc., or other smartphone operating systems, or specialized robot operating systems, such asRSS A2.

The cloud 720 provides cloud computing and/or cloud storagecapabilities. Cloud computing may provide Internet-based computing,whereby shared servers provide resources, software, and data tocomputers and other devices on demand. For example, the cloud 720 may bea cloud computing service that includes at least one server computingdevice, which may include a service abstraction layer and a hypertexttransfer protocol wrapper over a server virtual machine instantiatedthereon. The server computing device may be configured to parse HTTPrequests and send HTTP responses. Cloud computing may be a technologythat uses the Internet and central remote servers to maintain data andapplications. Cloud computing can allow users to access and useapplications 710 without installation and to access personal files atany computer with Internet access. Cloud computing allows for relativelymore efficient computing by centralizing storage, memory, processing andbandwidth. The cloud 720 can provide scalable, on-demand computingpower, storage, and bandwidth, while reducing robot hardwarerequirements (e.g., by freeing up CPU and memory usage). Robotconnectivity to the cloud 720 allows automatic data gathering of robotoperation and usage histories without requiring the robot 100 to returnto a base station. Moreover, continuous data collection over time canyield a wealth of data that can be mined for marketing, productdevelopment, and support.

Cloud storage 722 can be a model of networked computer data storagewhere data is stored on multiple virtual servers, generally hosted bythird parties. By providing communication between the robot 100 and thecloud 720, information gathered by the robot 100 can be securely viewedby authorized users via a web-based information portal.

The portal 730 may be a web-based user portal for gathering and/orproviding information, such as personal information, home statusinformation, and robot status information. Information can be integratedwith third-party information to provide additional functionality andresources to the user and/or the robot 100. The robot systemarchitecture 700 can facilitate proactive data collection. For example,applications 710 executed on the computing device 310 may collect dataand report on actions performed by the robot 100 and/or a person or anenvironment viewed by the robot 100 (using the sensor system 400). Thisdata can be a unique property of the robot 100.

“Dense data” vs. “sparse data” and “dense features” vs. “sparsefeatures” are referred to herein with respect to spatial data sets.Without limiting or narrowing the meaning from that of how those skilledin the art would interpret such terms to mean, “dense” vs. “sparse”generally means many data points per spatial representation vs. few datapoints, and specifically may mean:

(i) in the context of two-dimensional image data or three-dimensional“images” including two-dimensional data and range, “dense” image dataincludes image data substantially fully populated with pixels, orcapable of being rasterized to pixels with substantially no lossesand/or artifacting from the original image capture (includingsubstantially uncompressed, raw, or losslessly compressed images), whilea “sparse” image is one where the image is quantized, sampled, lossycompressed, vectorized, segmented (e.g., into superpixels, nodes, edges,surfaces, interest points, voxels), or otherwise materially reduced infidelity from the original capture, or must be interpolated in beingrasterized to pixels to re-represent an image;

(ii) in the context of two-dimensional or three-dimensional features,“dense features” may be features that are populated in a substantiallyunconstrained manner, to the resolution of the detection approach, allthat can be detected and recorded, and/or features that are recognizedby detectors recognized to collect many features (HOG, wavelets) over asub-image; “sparse features” may be purposefully constrained in number,in the number of feature inputs, lateral inhibition, and/or featureselection, and/or may be recognized by detectors recognized to identifya limited number of isolated points in an image (Harris corner, edges,Shi-Tomasi).

With respect to three-dimensional environment structure, the robot 100may acquire images, such as dense images 701, of a scene 10 about therobot 100 while maneuvering about a work surface 5. In someimplementations, the robot 100 uses a camera 320 and/or an image sensor450 (e.g., volumetric point cloud imaging device) for obtaining thedense images 701. The controller 500, which is in communication with thecamera 320 and/or the image sensor 450 may associate information withthe dense images 701 (e.g., mark-up or tag the dense images 701 withdata), such as accelerometer data traces, odometry data, and/or otherdata from the sensor system 400 along with timestamps. In some examples,the robot 100 captures a streaming sequence of dense images 701 andmarks the dense image sequence with mark-up data, providing a marked-updense image sequence. The cloud service 720 may process the receivedimage data 701 and return a processed data set to the robot controller500, which may issue drive commands to the drive system 200 based on thereceived processed data set for maneuvering about the scene 10.

The cloud service 720 may execute one of a variety of off-line methodsto process a stored image data set 703 into a dense three-dimensionalmap or model 705 of the scene 10 (environment) and then simplify thisdense three-dimensional map or model 705 into a two-dimensional heightmap 707, which can be a two-dimensional map with height data at eachpoint (e.g., similar to a two-dimensional topographical map). In someexamples, the two-dimensional height map 707 is a topographical maphaving X and Y coordinates with Z data. Each X,Y coordinate may have oneor more Z points (i.e., height data). Unlike the dense three-dimensionalmap, which may have numerous Z points (e.g., hundreds or thousands of Zpoints) for each X,Y coordinate, the two-dimensional height map 707 mayhave less than threshold number of Z points for each X,Y coordinate,such as between 2 and 20 (e.g., 10) points. A two-dimensional height map707 derived from a three-dimensional map of a table in a room may show afirst Z point for the bottom surface of a table top and a second Z pointfor the top surface of the table top for each X,Y coordinate along thetable. This information allows the robot 100 to determine if it can passunder the table top. By reducing the Z points from a dense data set of acontinuous range of Z points for each X,Y coordinate to a sparse dataset of a select number of Z points indicative of a detected objects 12,the robot 100 can receive a two-dimensional height map 707 having arelatively smaller size than the three-dimensional map used by the cloudservice 720. This, in turn, allows the robot 100 to store thetwo-dimensional height map 707 on local memory having a practical andcost effective size as compared to the scalable memory space availableto the cloud service 720. The robot 100 receives the two-dimensionalheight map 707 from the cloud 720, which provides the robot 100 andassociated controller 500 with navigational data for future work in thescene 10.

Additional methods and features of three-dimensional map datacompression are disclosed in “Multi-Level Surface Maps for OutdoorTerrain Mapping and Loop Closing” by R. Triebel, P. Pfaff and W.Burgard; IEEE/RSJ International Conference on Intelligent Robots andSystems, 2006, which is hereby incorporated by reference in itsentirety.

The cloud 720 provides the robot 100 with on-demand scaling of resources(e.g., computational, processing, memory, etc.) that may not otherwisebe practical or cost effective on the robot 100. For example, the cloud720 can provide scalable cloud storage 722 that scales up to a firstsize for storing and/or processing a relatively large amount of data701, which may only be used for a short period of time and thendiscarded, and then scaled back down to a second size. Moreover, thecloud 720 can provide computer processing power for executing relativelycomplex computations or “brute force” algorithms that might nototherwise be possible on the robot. By displacing computer processingpower and memory to a scalable cloud 720, the robot 100 can use acontroller 500 having relatively less computing power and memory, thusproviding a cost effective solution. Moreover, the robot 100 may executereal-time tasks (on the controller 500 or the web pad 310), such asobstacle avoidance, while passing non-real-time or non-time-sensitivetasks to the cloud 720 for processing and later retrieval.

The cloud 720 may execute one or more filters (e.g., a BundleAdjustment, RANSAC, Expectation Maximization, SAM or other 3D structuralestimation algorithms) for processing the stored image data set 703 intoa 3D representation. Once processed and a dense three-dimensional map705 has been created or updated, the image data set 703 can be discardedfrom the cloud storage 722, freeing up resources and allowing the cloud720 to scale accordingly. As a result, the robot 100 needs neither theon-board storage nor the processing to handle the storage and processingof the image data set 703, due to the use of cloud based resources. Thecloud 720 may return processed navigational data 701 or a map 707 (e.g.,a compressed two-dimensional height map) to the robot 100, which it canthen use for relatively simpler localization and navigation processing.

Additional methods and features of three-dimensional reconstruction aredisclosed in “3D Models from Extended Uncalibrated Video Sequences:Addressing Key-frame Selection and Projective Drift” by J. Repko and M.Pollefeys; Fifth International Conference on three-dimensional DigitalImaging and Modeling, 2005, which is hereby incorporated by reference inits entirety.

Referring to FIGS. 8A and 8B, in some circumstances, the robot 100receives an occupancy map 800 of objects 12 in a scene 10 and/or worksurface 5, or the robot controller 500 produces (and may update) theoccupancy map 800 based on image data and/or image depth data receivedfrom an image sensor 450 (e.g., the second three-dimensional imagesensor 450 b) over time. SLAM is a technique that may be used by therobot 100 to build up an occupancy map 800 within an unknown environmentor scene 10 (without a priori knowledge), or to update an occupancy map800 within a known environment (with a priori knowledge from a givenmap), while at the same time keeping track of its current location.

The controller 500 may communicate the occupancy map 800 to thetelepresence software application 601 for displaying a map 622 in theuser interface 605. The user interface map 622 may be derived partly orwholly from the occupancy map 800. Moreover, referring also to FIG. 7,the telepresence software application 601 may receive periodic updatesof the occupancy map 800 via the cloud service 720. For example, thecloud service 720 may provide with telepresence software application 601with the dense three-dimensional map or model 705 of the scene 10 aboutthe robot 100 and/or the simplified two-dimensional height map 707 forgenerating the user interface map 622. In additional examples, the cloudservice 720 provides the user interface map 622 to the telepresencesoftware application 601 based on the dense three-dimensional map ormodel 705 or the two-dimensional height map 707.

Referring again to FIGS. 8A and 8B, maps 800 can be used to determine alocation within an environment 10 and to depict an environment forplanning and navigation. The maps 800 support the assessment of actuallocation by recording information obtained from a form of perception andcomparing it to a current set of perceptions. The benefit of a map 800in aiding the assessment of a location increases as the precision andquality of the current perceptions decrease. Maps 800 generallyrepresent the state at the time that the map 800 is provided orproduced. This is not necessarily consistent with the state of theenvironment at the time the map 800 is used. Other localizationtechniques include monocular visual SLAM (MonoSLAM) and implementationsusing an extended Kalman filter (EKF) for MonoSLAM solutions.

The controller 500 may execute a scale-invariant feature transform(SIFT) to detect and describe local features in captured images. For anyobject 12 in an image, interesting points on the object 12 can beextracted to provide a “feature description” of the object 12. Thisdescription, extracted from a training image, can then be used toidentify the object 12 when attempting to locate the object 12 in a testimage containing many other objects. To perform reliable recognition, itis important that the features extracted from the training image bedetectable even under changes in image scale, noise and illumination.Such points usually lie on high-contrast regions of the image, such asobject edges. For object recognition and detection, the robot 100 mayuse a SIFT to find distinctive key points that are invariant tolocation, scale and rotation, and robust to affine transformations(changes in scale, rotation, shear, and position) and changes inillumination. In some implementations, the robot 100 captures multipleimages (using the camera 320 and/or image sensor 450) of a scene 10 orobject 12 (e.g., under different conditions, from different angles,etc.) and stores the images, such as in a matrix. The robot 100 canaccess the stored images to identify a new image by comparison, filter,etc. For example, SIFT features can be obtained from an input image andmatched to a SIFT feature database obtained from training images(captured previously). The feature matching can be done through aEuclidean-distance based nearest neighbor approach. A Hough transformmay be used to increase object identification by clustering thosefeatures that belong to the same object and reject the matches that areleft out in the clustering process. A speeded up robust feature (SURF)may be a robust image detector and descriptor.

In addition to localization of the robot 100 in the scene 10 (e.g., theenvironment about the robot 100), the robot 100 may travel to otherpoints in a connected space (e.g., the work surface 5) using the sensorsystem 400. The robot 100 may include a short range type of image sensor450 a (e.g., mounted on the underside of the torso 140, as shown inFIGS. 1 and 3) for mapping a nearby area about the robot 110 anddiscerning relatively close objects 12, and a long range type of imagesensor 450 b (e.g., mounted on the head 160, as shown in FIGS. 1 and 3)for mapping a relatively larger area about the robot 100 and discerningrelatively far away objects 12. The robot 100 can use the occupancy map800 to identify known objects 12 in the scene 10 as well as occlusions16 (e.g., where an object 12 should or should not be, but cannot beconfirmed from the current vantage point). The robot 100 can register anocclusion 16 or new object 12 in the scene 10 and attempt tocircumnavigate the occlusion 16 or new object 12 to verify the locationof new object 12 or any objects 12 in the occlusion 16. Moreover, usingthe occupancy map 800, the robot 100 can determine and track movement ofan object 12 in the scene 10. For example, the image sensor 450, 450 a,450 b may detect a new position of the object 12 in the scene 10 whilenot detecting a mapped position of the object 12 in the scene 10. Therobot 100 can register the position of the old object 12 as an occlusion16 and try to circumnavigate the occlusion 16 to verify the location ofthe object 12. The robot 100 may compare new image depth data withprevious image depth data (e.g., the map 800) and assign a confidencelevel of the location of the object 12 in the scene 10. The locationconfidence level of objects 12 within the scene 10 can time out after athreshold period of time. The sensor system 400 can update locationconfidence levels of each object 12 after each imaging cycle of thesensor system 400. In some examples, a detected new occlusion 16 (e.g.,a missing object 12 from the occupancy map 800) within an occlusiondetection period (e.g., less than 10 seconds) may signify a “live”object 12 (e.g., a moving object 12) in the scene 10.

In some implementations, a second object 12 b of interest, locatedbehind a detected first object 12 a in the scene 10, may be initiallyundetected as an occlusion 16 in the scene 10. An occlusion 16 can be anarea in the scene 10 that is not readily detectable or viewable by theimage sensor 450, 450 a, 450 b. In the example shown, the sensor system400 (e.g., or a portion thereof, such as image sensor 450, 450 a, 450 b)of the robot 100 has a field of view 452 with a viewing angle θ_(V)(which can be any angle between 0 degrees and 360 degrees) to view thescene 10. In some examples, the image sensor 450 includesomni-directional optics for a 360 degree viewing angle θ_(V) while inother examples, the image sensor 450, 450 a, 450 b has a viewing angleθ_(V) of less than 360 degrees (e.g., between about 45 degrees and 180degrees). In examples, where the viewing angle θ_(V) is less than 360degrees, the image sensor 450, 450 a, 450 b (or components thereof) mayrotate with respect to the robot body 110 to achieve a viewing angleθ_(V) of 360 degrees. In some implementations, the image sensor 450, 450a, 450 b or portions thereof, can move with respect to the robot body110 and/or drive system 200. Moreover, in order to detect the secondobject 12 b, the robot 100 may move the image sensor 450, 450 a, 450 bby driving about the scene 10 in one or more directions (e.g., bytranslating and/or rotating on the work surface 5) to obtain a vantagepoint that allows detection of the second object 12 b. Robot movement orindependent movement of the image sensor 450, 450 a, 450 b, or portionsthereof, may resolve monocular difficulties as well.

A confidence level may be assigned to detected locations or trackedmovements of objects 12 in the working area 5. For example, uponproducing or updating the occupancy map 800, the controller 500 mayassign a confidence level for each object 12 on the map 800. Theconfidence level can be directly proportional to a probability that theobject 12 is actually located in the working area 5 as indicated on themap 800. The confidence level may be determined by a number of factors,such as the number and type of sensors used to detect the object 12. Forexample, the contact sensor 430 may provide the highest level ofconfidence, as the contact sensor 430 senses actual contact with theobject 12 by the robot 100. The image sensor 450 may provide a differentlevel of confidence, which may be higher than the proximity sensor 430.Data received from more than one sensor of the sensor system 400 can beaggregated or accumulated for providing a relatively higher level ofconfidence over any single sensor.

Odometry is the use of data from the movement of actuators to estimatechange in position over time (distance traveled). In some examples, anencoder is disposed on the drive system 200 for measuring wheelrevolutions, therefore a distance traveled by the robot 100. Thecontroller 500 may use odometry in assessing a confidence level for anobject location. In some implementations, the sensor system 400 includesan odometer and/or an angular rate sensor (e.g., gyroscope or the IMU470) for sensing a distance traveled by the robot 100. A gyroscope is adevice for measuring or maintaining orientation, based on the principlesof conservation of angular momentum. The controller 500 may use odometryand/or gyro signals received from the odometer and/or angular ratesensor, respectively, to determine a location of the robot 100 in aworking area 5 and/or on an occupancy map 800. In some examples, thecontroller 500 uses dead reckoning. Dead reckoning is the process ofestimating a current position based upon a previously determinedposition, and advancing that position based upon known or estimatedspeeds over elapsed time, and course. By knowing a robot location in theworking area 5 (e.g., via odometry, gyroscope) as well as a sensedlocation of one or more objects 12 in the working area 5 (via the sensorsystem 400), the controller 500 can assess a relatively higherconfidence level of a location or movement of an object 12 on theoccupancy map 800 and in the working area 5 (versus without the use ofodometry or a gyroscope).

Odometry based on wheel motion can be electrically noisy. The controller500 may utilize scan matching in conjunction with or in place of wheelodometry. The use of scan matching may improve accuracy and/or reducethe computational burden. In such an embodiment, two partial mapsobtained using LIDAR and/or other mapping methods may be merged into asingle map. The two or more partial maps may be merged using a knownscanning location. Alternatively, two or more partial maps may be mergedusing geometrical features of the partial scans. The controller 500 mayreceive image data from the image sensor 450 of the environment or scene10 about the robot 100 for computing robot motion, independently ofwheel based odometry of the drive system 200, through visual odometry.Visual odometry may entail using optical flow to determine the motion ofthe image sensor 450. The controller 500 can use the calculated motionbased on imaging data of the image sensor 450 for correcting any errorsin the wheel based odometry, thus allowing for improved mapping andmotion control. Visual odometry may have limitations with low-texture orlow-light scenes 10, if the image sensor 450 cannot track featureswithin the captured image(s).

Other details and features on odometry and imaging systems, which may becombinable with those described herein, can be found in U.S. Pat. No.7,158,317 (describing a “depth-of field” imaging system), and U.S. Pat.No. 7,115,849 (describing wavefront coding interference contrast imagingsystems), the contents of which are hereby incorporated by reference intheir entireties.

Referring to FIGS. 8C and 8D, when a robot is new to a building that itwill be working in, the robot may need to be shown around or providedwith a map of the building (e.g., room and hallway locations) forautonomous navigation. For example, in a hospital, the robot may need toknow the location of each patient room, nursing stations, etc. In someimplementations, the robot 100 receives a plan view map 810, such as theone shown in FIG. 8C, and can be trained to learn the plan view map 810.For example, while leading the robot 100 around the building, the robot100 may record specific locations corresponding to locations on the planview map 810. The robot 100 may display the plan view map 810 on the webpad 310 and when the user takes the robot 100 to a specific location,the user can tag that location on the plan view map 810 (e.g., using atouch screen or other pointing device of the web pads 310). The user maychoose to enter a label for a tagged location, like a room name or aroom number. At the time of tagging, the robot 100 may store the tag,with a point on the plan view map 810 and a corresponding point on arobot map 820, such as the one shown in FIG. 8D. As illustrated, therobot map 820 may be a two-dimensional plan view map similar to the planview map 810. In alternative embodiments, the robot map 820 may be athree-dimensional map including a ground level corresponding to atwo-dimensional plan view map similar to the plan view map 810.

Using the sensor system 400, the robot 100 may build the robot map 820as it moves around. For example, the sensor system 400 can provideinformation on how far the robot 100 has moved and a direction oftravel. The robot map 820 may include fixed obstacles in addition to thewalls provided in the plan view map 810. The robot 100 may use the robotmap 820 to execute autonomous navigation. In the robot map 820, the“walls” may not look perfectly straight, for example, due to detectedpacking crates along the wall in the corresponding hallway and/orfurniture detected inside various cubicles. Moreover, rotational andresolution differences may exist between the plan view map 810 and therobot map 820.

Referring to FIG. 8E, in some implementations, the telepresence softwareapplication 601 displays a tagging view 660 that allows the user toplace tags 662 on the plan view map 810. The plan view map 810 may bethe same map as that displayed by the plan view map window 620 or it maybe a different map used internally for navigation purposes.

The user, a remote terminal, and/or the robot may insert tags 662 ontospecific locations of the plan view map 810 and/or robot map 820 to markmap locations with information, such as driving hazards, obstacles,robot aids, etc. For example, the user may drag-and-drop tags 662 ontospecific location of the plan view map 810. As is described herein, thetags may include tag coordinates associated with a point or a region,tag information defining the purpose of the tag, type of the tag, natureof the tag, instructions for the user and/or robot related to the tag,and/or other information relevant to the tag, and finally, the tag mayinclude a tag annotation comprising a two-dimensional and/orthree-dimensional graphic or text corresponding to the tag. An exampleof a tag annotation is an octagonal red stop sign associated with a tagcontaining tag information indicating an area that a robot should notenter. A tag annotation may be human and/or machine interpretable. Tagcoordinates may be points, lines, plans, surfaces, volumes, and/or 2.5Dor hybrid surfaces. Tags may be formed as data structures having anynumber of additional fields and/or parameters. For example, a taginclude fields associated with time, scheduling, spatial coordinates,and/or triggers for predetermined functions.

The term annotation, as used herein, includes text, words, or otherverbal representations. Accordingly, the tag annotation may be apicture, graphical image, a pictograph, a hieroannotation, a non-verbalsymbol. In addition, a tag annotation may be a word, a letter, a phrase,or other text form. For example, a tag associated with a nurses stationmay include a tag annotation comprising a textual representation of thenurses name. The textual representation of the nurses name could betwo-dimensional text, or it could be a three-dimensional text.Alternatively, the tag annotation associated with the nurse's stationcould be a large letter N, or a symbol representative of a nurse'sstation (e.g., a nurses hat or nursing symbol).

The tags 662 may include a wireless local area network (WLAN) warm tag662 a denoting an area having relatively good signal reception and aWLAN cold tag 662 b denoting an area having relatively poor signalreception. The robot 100 may use this information to navigate from onelocation to another through an area having relatively good wirelesssignal reception, while avoiding areas having relatively poor wirelesssignal reception.

A low traffic tag 662 c denotes an area having relatively low traffic(person and/or robot). The robot 100 may select a travel path though anarea having a relatively low traffic volume, rather than through an areahaving a relatively high traffic volume. Moreover, if the robot 100 musttravel through a high traffic area, the robot 100 may execute one ormore specific object detection obstacle avoidance (ODOA) behaviors tonavigate the area successfully without collisions.

A dock tag 662 d denotes a location of a robot docking station. A lowbattery event may signal the controller 500 to seek recharging. Therobot 100 may use the map location tagged with a dock tag 662 d tolocate a robot docking station for recharging. For example, by applyinga resolved distortion between the plan view map 810 and the robot map820 (FIGS. 8C and 8D), the robot 100 can determine a corresponding robotmap location 824 corresponding to the tagged layout map location 814 tonavigate to that tagged location to dock with a robot docking station.Resolving the distortion may include determining a distortion betweentwo maps that use the same coordinate system. The robot map 820 and theplan view map 810 may both be two-dimensional and as such, determining adistortion may not require determining a coordinate transformationbetween different dimensions.

Some tags 662 may be used to indicate obstacles or special traversalareas. For example, a glass tag 662 e indicates the location of a glasswall, window, or door. The robot 100 may use this information to avoidthe tagged glass structures, since infrared proximity sensors may notdetect them. A ramp tag 662 f indicates the location of a floor ramp.For a distance, the robot 100 may detect the ramp as an obstacle, sinceit may appear to have a vertical height greater than a thresholdtraversal height. When approaching a tagged ramp, the robot 100 mayexecute a ramp or traversal behavior to successfully negotiate the ramp.A tight tag 662 g indicates the location of a relatively narrow corridoror throughway. The robot 100 may avoid such areas, so as to avoid anyconfining situations.

A slow tag 662 h indicates a location or area in which the robot 100drives relatively slowly. This location or area may coincide with a hightraffic area. An avoid tag 662 i denotes a location or area that therobot 100 should avoid (i.e., not drive through). In some embodiments,an avoid tag 622 i may be operation mode-dependent. For example, theavoid tag 622 i may be applicable only when the robot is operating in afully autonomous mode. During teleoperation, the avoid tag 622 i may beeffectively ignored by the robot. An operating room user interface (ORUI) tag 662 j indicates the location or area of a hospital operatingroom. The robot 100 may use this tag to find the operating room toprovide telepresence support and/or to display a specific user interface(e.g., an OR UI) upon entering the OR area. A training tag 662 k can beused to mark general locations, such as hallways and rooms, to train therobot 100 to learn its environment 10.

A manual elevator tag 662l indicates the location of an elevator wherethe robot 100 should allow a user to aid the robot's traversal into/outof the elevator. Manual elevator negotiation may be based on remote userpiloting or robot-local user guidance. For remote user piloting, aremote user provides drive commands to the robot 100 (e.g., using ajoystick). For robot-local user guidance, a person adjacent to the robot100 may physically touch the robot 100 and, in response to thosetouches, the robot 100 moves accordingly. Features regarding robotresponsiveness to user touching combinable with those described hereincan be found in application Ser. No. 13/032,390, filed on Feb. 22, 2011,which is hereby incorporated by reference in its entirety.

An auto elevator tag 662 m indicates the location of an elevator thatthe robot 100 may negotiate (into and out of) autonomously. The robot100 may execute a threshold traversal behavior 512 d (FIG. 5) to enterinto and drive out of the elevator, so as to avoid tipping. Featuresregarding robot responsiveness to user touching combinable with thosedescribed herein can be found in application serial numberPCT/US11/59910, filed on Nov. 9, 2011, which is hereby incorporated byreference in its entirety.

A keep right tag 662 n indicates a map location or area in which therobot 100 should keep to the right. A user may place this tag alongcertain corridors, such as high traffic corridors. In response to thekeep right tag 662 n, the robot 100 may execute a wall followingbehavior to stay along the wall while driving in the tagged area.

After map training, when a user wants to send the robot 100 to alocation, the user can either refer to a label/tag 622 (e.g., enter alabel or tag into a location text box displayed on the web pad 310) orthe robot 100 can display the plan view map 810 to the user on the webpad 310 and the user may select the location on the plan view map 810.If the user selects a tagged layout map location 814, the robot 100 caneasily determine the corresponding robot map location 824 on the robotmap 820 and can proceed to navigate to the selected location 814.

In some implementations, the robot controller 500 may execute a firstbehavior 512 while maneuvering about a first area and then execute asecond behavior 512 while maneuvering about a second area associatedwith a tag having an associated robot behavior modifier. For example,while executing a person follow behavior 512 b, the robot controller mayeither cease execution of that behavior 512 b or concurrently execute athreshold traversal behavior 512 d upon reaching a map location 814tagged with a ramp tag 662 f or an auto elevator tag 622 m.

If the selected location on the plan view map 810 is not a taggedlocation 814, the robot 100 determines a corresponding location 824 onthe robot map 820. In some implementations, the robot 100 computes ascaling size, origin mapping, and rotation between the plan view map 810and the robot map 820 using existing tagged locations, and then appliesthe computed parameters to determine the robot map location (e.g., usingan affine transformation or coordinates).

The robot map 820 may not be the same orientation and scale as the planview map 810. Moreover, the layout map may not be to scale and may havedistortions that vary by map area. For example, a plan view map 810created by scanning a fire evacuation map typically seen in hotels,offices, and hospitals is usually not drawn to scale and can even havedifferent scales in different regions of the map. The robot map 820 mayhave its own errors. For example, locations on the robot map 820 mayhave been computed by counting wheel turns as a measure of distance, andif the floor was slightly slippery or turning around corners causedextra wheel turns, inaccurate rotation calculations may cause the robot100 to determine inaccurate locations of mapped objects.

A method of mapping a given point 814 on the plan view map 810 to acorresponding point 824 on the robot map 820 may include using existingtagged points 812 to compute a local distortion (in the sametwo-dimensional coordinate system) between the plan view map 810 and therobot map 820 in a region (e.g., within a threshold radius) containingthe layout map point. The method further includes applying a distortioncalculation to the layout map point 814 in order to find a correspondingrobot map point 824. The reverse can be done if you are starting with agiven point on the robot map 820 and want to find a corresponding pointon the plan view map 810, for example, for asking the robot for itscurrent location.

Any of a wide variety of tag schemas and data structures may be used.For example, tags may contain attributes in the form of key-value pairsthat specify a purpose of the tag, tag parameters, and tag attributes(generally “tag information”). Table 1 below provides a specificexample.

TABLE 1 Field name Data type Description tagId integer the ID of the tagentry in the tag table described above name text the parameter namevalue text the parameter value

Tags associated with regions may have attributes associated with themthat specify their purpose and specify parameters that influencebehaviors associated with the region. These key-value pairs may bestored using a data structure similar to the example in Table 2 below:

TABLE 2 Field name Data type Description regionId integer the ID of theregion entry in the region table described above name text the parametername value text the parameter value

The data structure for each tag may include tag coordinates and taginformation, which may include a tag annotation (such as atwo-dimensional and/or three-dimensional graphical representation of thetag). Table 3 below provides a specific example of a data structure fora tag.

TABLE 3 Field name Data type Description id integer globally uniqueidentifier for the tag in the database mapId integer the identifier ofthe robot map to which the tag belongs timestamp float a timestamprepresenting when the tag was created poseX float the X coordinate ofthe tag in the robot map's coordinate system poseY float the Ycoordinate of the tag in the robot map's coordinate system poseZ floatthe Z coordinate of the tag in the robot map's coordinate system poseXrfloat the tag's rotation about the X axis poseYr float the tag'srotation about the Y axis poseZr float the tag's rotation about the Zaxis name text a human-readable identifier for the tag annotation imagea 2D and/or 3D graphical representation

As described herein, a tag may be associated with a region, rather thana specific point, on a map. There may be a many-to-one relationshipbetween tag information and a tag. A specific example of a datastructure for a tag associated with a region is provided below in Table4.

TABLE 4 Field name Data type Description id integer globally uniqueidentifier for the region in the database mapId integer the identifierof the robot map to which the region belongs timestamp float a timestamprepresenting when the region was created poseX float the X coordinate ofthe region's centroid in the same coordinate system as the robot mapposeY float the Y coordinate of the region's centroid in the samecoordinate system as the robot map poseZ float the Z coordinate of theregion's centroid in the same coordinate system as the robot map poseXrfloat the region's rotation about the X axis poseYr float the region'srotation about the Y axis poseZr float the region's rotation about the Zaxis name text a human-readable identifier for the region annotationimage a 2D and/or 3D graphical representation

In some examples, the geometry of regions may be broken into thecomponents of their centroid and offsets from the centroid to allow forthe position and rotation of many objects to be updated quickly. WhenCPU resources permit, the bounding box of the final coordinates (thepolygon points relative to the centroid, transformed by the centroid'spose into the map's coordinate system) can be indexed using an R*-treeor similar data structure for fast lookup based on geometricconstraints. The points comprising the region's polygon may be stored inclockwise (or counter-clockwise) order to facilitate point-in-polygontests based on ray-tracing algorithms.

As an example, a tag indicating a region that is a slow zone may have adata structure as provided below in Table 5.

TABLE 5 Name Value type speedZone subtype explicit maxXVelocity 0.75maxYVelocity 0.0 maxThetaVelocity 0.75

Table 5 illustrates an example of a slow-zone tag in which the speedlimits are explicitly set based on values associated with the regionitself. Alternatively, a region may be defined in such a manner that arobot may interpret the region as a slow zone. For example, in Table 6below, a robot may interpret a region defined as an intersection as aslow zone and reduce its speed to a predetermined velocity.

TABLE 6 Name Value type speedZone subtype intersection

Tags associated with points or regions on a map may include tagcoordinates, tag information, and tag annotations of any of a widevariety. Additionally, the tags may be implemented using any of a widevariety of data types, including those illustrated above in Tables 1-6.The terms tag information and tag annotation are referred herein asseparate elements. However, according to various embodiments, a tagannotation may be a part of the tag information. Specifically, a datastructure may or may not include a distinct field for the tagannotation. Rather, a field in the data structure for the taginformation may incorporate the tag annotation.

FIG. 8F provides an exemplary arrangement 800 f of operations foroperating the robot 100 to navigate about an environment using the planview map 810 and the robot map 820. The operations include receiving 802f a plan view map 810 corresponding to an environment of the robot 100,moving 804 f the robot 100 in the environment to a layout map location812 on the plan view map 810, recording 806 f a robot map location 822on a robot map 820 corresponding to the environment and produced by therobot 100, determining 808 f a distortion (two-dimensional) between therobot map 820 and the plan view map 810 using the recorded robot maplocations 822 and the corresponding layout map locations 812, andapplying 810 f the determined distortion to a target layout map location814 to determine a corresponding target robot map location 824, thusallowing the robot to navigate to the selected location 814 on the planview map 810. In some implementations, the operations includedetermining a scaling size, origin mapping, and rotation between theplan view map 810 and the robot map 820 using existing tagged locationsand resolving a robot map location corresponding to the selected targetlayout map location 814. The operations may include applying an affinetransformation to the determined scaling size, origin mapping, androtation to resolve the robot map location. Any of the above operationsmay be repeated any number of times in order to increase accuracy and/orefficiency. For example, moving 804 f the robot 100 in the environmentand recording 806 f a robot map location may be repeated multiple timesin order to gather sufficient correlation points for the subsequenttransformations and calculations between the layout map and the robotmap.

Other details and features combinable herewith can be found in PCTapplication Serial No.: PCT/US11/60935, filed on Nov. 16, 2011, which ishereby incorporated by reference in its entirety.

Referring to FIGS. 9A and 9B, in some implementations, the teleoperationsoftware application 601 displays a hybrid three-dimensional image map622 b (hybrid map) in the map window 620. The hybrid map 622 b may be acombination of the remote view 612 displayed in the remote video feedwindow 610 and a plan view map 810, such as the two-dimensional,top-down map 622 a displayed in the plan view map window 620 (FIG. 6D).FIG. 9A illustrates a remote video view 612 that a user may see when therobot 100 is positioned in a hallway. FIG. 9B illustrates a hybrid map622 b in which the plan view map 810 is partially overlaid and modifiedto fit the remote view 612, indicating the room numbers and/or roomtypes of the areas in the field of view of the robot 100. When viewing alive video feed, the user may place the cursor over the window and startmoving the scroll wheel upward. During the transitional process, theperspective video view (from the camera 320 on the robot 100) isprogressively transitioned between the remote video view 612 and the map622. The map 622 is fully warped at the start of the transition to mapthe perspective remote video view 612, and gradually reverts to itsunwarped view at the end of the transition. So if the mouse wheel is 30%up, then the user sees a dissolved image which contains 70% video and30% map, and the video portion is 30% de-warped, while the map is 70%warped. This implementation allows for a single view to fluidlyrepresent both a perspective live remote video view 612 and a map 622.

To provide the hybrid map 622 b, the teleoperation software application601 may determine a distortion (between two-dimensional coordinates andthree-dimensional coordinates) between the remote view 612 and the planview map 810 using recorded robot map locations 822 of a robot map 820and corresponding layout map locations 812 and then applying thedetermined distortion to fit the plan view map 810 to the remote view612. In some implementations, determining the distortion includesdetermining a scaling size, origin mapping, and rotation between theplan view map 810 and the remote view 612, for example, by applying anaffine transformation to the determined scaling size, origin mapping,and rotation. Determining a distortion between a two-dimensional planview map and a three-dimensional video feed may include determining acoordinate transformation between the disparate coordinate systems.

Referring to FIGS. 6D and 10A-10E, in some implementations, the userinterface 605 provides a look-ahead command 624 that causes the displayof a rendered look-ahead view 612 a in the map window 620, a dedicatedseparate window, or some other window. While driving the robot 100, theuser may invoke a look-ahead command 624, which causes the robot 100 tostop moving physically, while the teleoperation software application 601generates and displays a rendered look-ahead view 612 a providing aperspective view of a proposed robot drive path as if the robot 100 werecontinuing to move along its drive path. This may be accomplished byusing the map data, such as location of walls, and constructing aperspective “virtual reality” view based on the virtual location of therobot 100. For example, the telepresence software application 601 mayuse the plan view map 810, the robot map 820, and/or stored image data701 (FIG. 7) to construct the look-ahead view 612 a. For robot systemsusing a cloud computing service 720, such as the example shown in FIG.7, the telepresence software application 601 and optionally the robot100 may communicate with the cloud computing service 720, which mayconstruct the look-ahead view 612 a based on stored image data 701, thethree-dimensional map 705, and/or the two-dimensional height map 707 (oralternatively a 2.5D hybrid map) and then provide the look-ahead view612 a for rendering in the map window 620. This implementation allowsthe telepresence software application 601 to leverage the scalablecomputer processing and data storage capability of the cloud service(e.g., the cloud service 720 can elastically scale up to process thedata and then scale down afterwards), thus reducing a processing andmemory requirement for a computing device executing the telepresencesoftware application 601.

FIG. 10A illustrates an exemplary remote view 612 of the remote videofeed window 610 of the telepresence software application 601. FIG. 10Billustrates a complementary map 622 displayed in the map window 620. Themap 622 provides the current location of the robot 100 as denoted by therobot icon 650 along with a camera field of view 322 for a robot camera320. FIGS. 10C and 10E provide exemplary look-ahead views 612 adisplayed in the remote video feed window 610. The remote video feedwindow 610 may continue to display the remote view 612 from the robotcamera 320 in a picture-in-picture window, for example, placed in acorner of the remote video feed window 610. FIGS. 10D and 10F provideexemplary maps 622 displayed in the map window 620. While executing thelook-ahead command, the telepresence software application 601 may renderthe robot icon 650 at the robot's current location along with therobot's camera field of view 322. In addition or alternatively, thetelepresence software application 601 may render in the plan view mapwindow 620 a virtual robot icon 650 a moving along a look-ahead pathalong with a projected look-ahead camera field of view 322 a.

In some implementations, as the user drives the robot 100 along acorridor using a joystick in communication with the telepresencesoftware application 601, the user may invoke the look-ahead command 624(e.g., by selecting a corresponding button on the user interface 605 orthe joystick). For example, at a location 50 feet away from a turn inthe corridor, the user may invoke the look-ahead command 624, causingthe generation of a look-ahead view 612 a and stopping further movementof the robot 100 along the corridor. The user may continue, however, tovirtually move the robot 100 in a look-ahead mode. The user interface605 may display the look-ahead view 612 a (e.g., a three-dimensionalmodel) of the same corridor at the same position. As the user drivesforward in the look-ahead mode, continues 50 feet, turns left, andcontinues driving, the user can see the location of rooms and othercorridors along the way in the three-dimensional model/look-ahead view612 a. In some examples, for the first 30 feet of “virtual” driving, thetelepresence software application 601 may display a blend of the actualview (from the stationary physical robot, further magnified andperspective-warped to match the virtual location) and thethree-dimensional model/look-ahead view 612 a.

FIG. 10G provides an exemplary arrangement 1000 of operations forexecuting a look-ahead mode of the telepresence software application601. The operations include initiating 1002 the look-ahead mode (alsoreferred to as the flythrough mode) and checking 1004 an actuallocalization of the robot 100, such as a current pose and/or coordinatesof the robot 100. The robot 100 may determine its localization based onreceived sensor signals of its sensor system 400 and then forward thelocalization to the telepresence software application 601 and/or a cloudcomputing service 720. The operations further include creating 1006 avirtual localization and/or pose of the robot 100. The telepresencesoftware application 601 or the cloud computing service 720 may use adynamics model 570 (FIG. 5) associated with the robot 100 and image data701 (FIG. 7) (e.g., volumetric point cloud image data) to generate thevirtual localization and/or pose. The operations may include accessing1008 three-dimensional rendering data corresponding to the determinedvirtual robot localization and generating 1010 a three-dimensionalrendering of the robot 100 and/or local environment about the robot 100.This may entail accessing image data 701 (e.g., volumetric point cloudimage data) and/or the three-dimensional map 705 stored locally orremotely in cloud storage 722 to construct the local three-dimensionalmodel/look-ahead view 612 a, which may be displayed by the telepresencesoftware application 601 in the remote video feed window 610. Moreover,this may entail generating a three-dimensional model of the robot 100illustrated by the virtual robot icon 650 a and the projected look-aheadcamera field of view 322 a in the map window 620. The operations mayinclude updating 1012 the displayed look-ahead view 612 a or a firstperson point of view (POV) and updating a virtual localization/pose ofthe robot 100 as the robot 100 virtually maneuvers about in thelook-ahead mode. Steps 1008-1014 can be repeated (e.g., on a periodicbasis) until terminating 1016 the look-ahead/flythrough mode.

Referring to FIG. 11A, in some implementations, the user interface 605of the telepresence software application 601 displays a remotenavigation view 612 b in the remote video feed window 610 (or anotherwindow). The remote navigation view 612 b may have a navigable area 616rendered over the live video feed of the remote view 612. The user maytoggle between the remote view 612 and the remote navigation view 612 b.The navigable area 616 may be determined based on the plan view map 810and/or the robot map 820. The navigable area 616 may be shown as abounded area with a field of view 832 of the robot camera(s) 320, 450,excluding obstacles. Moreover, the navigable area 616 may be filled witha color or other signal that communicates to a user that the navigablearea is free of obstacles or other impediments.

Navigable area on the layout map may be highlighted based on informationin the robot's internal obstacle map. In one embodiment, the navigablearea may be identified as white-colored pixels in the image. The robotmay return its position on the robot map and the position andorientation of the 3D depth camera. A processor may use the robotposition and the kinematic state (e.g., pan and tilt angles) of the headcamera to determine which pixels on the video screen represent theground plane. That is, the processor may utilize the robot position andthe perspective of the video feed to calculate the coordinates for eachground level pixel on the video feed. The white-colored pixelsdesignating the navigable areas may then be overlaid on the ground levelpixels on the video feed. Accordingly, the robot and/or a usercontrolling the robot could identify navigable areas by following thewhite-colored pixels. In alternative embodiments, any color pixel orother identifying mark could be used. Alternative data structures ormarks could be used in place of white-colored pixels. Specifically, fromthe robot's POV the coordinates of ground level pixels that arenavigable could be tagged in any of a wide variety of ways, so long asthe robot is configured to recognize them.

The user may select a robot destination 618 in the navigable area 616,which causes the telepresence software application 601 to issue a drivecommand to the robot 100 to move to a location corresponding to theselected robot destination 618. In the example shown, the remote videofeed window 610 of the user interface 605 provides a remote navigationview 612 b of the robot 100 in a hospital room. The user selects a robotdestination 618 as a location in the room next to a patient bed. Thetelepresence software application 601 may use the plan view map 810,robot map 820, three-dimensional map 705, two-dimensional (or 2.5Dhybrid) height map 707, and/or stored image data 701 to resolve adistortion (within the same dimension and/or between dimensions) betweenthe selected robot destination 618 on the remote navigation view 612 band a corresponding robot map location 824. The telepresence softwareapplication 601 may then issue a drive command to the robot 100 tomaneuver autonomously or semi-autonomously to the robot map location824, using its sensor system 400 and behavior system 510 a to avoid anyobstacles, such as moving people.

In one example, a map may be returned from a robot API call as an image,such as a PNG, JPG, or TIFF image. The robot could process the image inorder to detect the pixels (e.g., black-colored pixels) that form theoutline of an obstacle in the image. A curve fitting algorithm could beused to process the pixels that form the outline of the obstacle. Theresulting curve(s) could then be used to generate an obstacle map.Additional processing may be done to further improve the obstacledetection and/or improve the accuracy of the curves fitting the detectedobstacle outlines. For example, if a curve closes and forms a shapesimilar to a circle, the obstacle map could simply use a circle as areplacement. A similar idea could be applied to shapes like rectanglesor ovals, people, faces, and/or those objects approximating a databaseof known object shapes from various perspectives.

The user interface 605 may provide a proximity sensor window 670 whichdisplays a proximity of obstacles within a sensor field of view 442, 452(e.g., within an three-dimensional imaging sensor field of view 452and/or a laser scanner field of view 442).

In some implementations, the user may mark a protected region/zone onthe remote video feed window 610 and/or the plan view map window 620(not shown), using an avoid tag 662, 662 i. A protected zone may betreated by the robot 100 as an object 12, and accordingly, protectedzones may be avoided during autonomous navigation. Protected zones canbe used to help create a wide berth around delicate equipment, or inorder to ensure that the robot avoids other areas. The user may place anavoid tag 662, 662 i on the plan view map 810 in the tagging view 660 oron the remote navigation view 612 b. Moreover, the user may place othertags 662 on the remote navigation view 612 b. The telepresence softwareapplication 601 may resolve a distortion between the remote navigationview 612 b and the plan view map 810 and/or robot map 820 and thenupdate the robot map 820 accordingly.

For example, determining a distortion between a plan view map and avideo feed may comprise creating a transformation mapping betweencoordinate points in any of the navigation view 612 b, the plan view map810, and/or the robot map 820. Similar to overlaying a restricted regionon a ground level in a video feed, the ground level of two-dimensionalmaps may be effectively coordinate-mapped onto a detected ground levelin a video feed provided by a robot.

FIG. 11B illustrates a flow chart of an exemplary arrangement 1100 ofoperations for a method of robot navigation (e.g., semi-autonomous) to aselected robot destination 618. The method includes identifying 1102 anavigable area 616 within a field of view 322, 442, 452 of the robot100. Identification of navigable areas 616 may be accomplished using thesensor system 400 of the robot 100. The method also includes visuallyindicating 1104 the navigable areas 616 on the user interface 605, forexample by displaying a bounded area (e.g., highlighted boundaries,filled with a color or pattern) on the remote navigation view 612 b. Themethod may include receiving 1106 a user selection of a robotdestination 618 and determining 1108 whether the robot destination 618is within the identified navigable areas 616. If the robot destinationis outside of identified navigable areas 616, the method includesprompting 1110 the user for a valid robot destination 618 within thenavigable area(s) 616. If the robot destination 618 is within identifiednavigable areas 616, the method may include determining 1112 a route tothe robot destination 618. This may entail determining a distortionbetween the remote navigation view 612 b and the robot map 820 and thenresolving a robot map location 824 corresponding to the selected robotdestination 618. The method includes allowing 1114 the robot 100 tonavigate (autonomously or semi-autonomously) to the robot destination618.

FIGS. 11C and 11D illustrate exemplary remote navigation views 612 bwhere the user selected a robot destination 618 either beyond anavigable area 616 or on an obstacle 1120 (either actual or perceived bythe robot 100). In the example shown in FIG. 11C, the user selected arobot destination 618 on a perceived obstacle 1120 a, a ramp 1122. Froma distance, the robot sensor system 400 may discern the ramp 1122 as anobstacle, because from a distance the ramp 1122 may have a perceivedheight above a threshold traversal height of the robot 100. Moreover,the robot behavior system 510 a may execute an ODOA behavior 512 c inresponse to sensor events raised due to sensor signals of the sensorsystem 400 indicating an obstacle having a height greater than thethreshold traversal height. Using the plan view map 810 and/or the robotmap 820, the robot 100 may determine that its local perception of theenvironment may be inaccurate, and that the ramp 1122 is not an actualobstacle, but is rather a perceived obstacle 1120 a.

Although the ramp 1122 is within the navigable area 616, thetelepresence software application 601 may determine the robotdestination 618 on the ramp 1122 is an unsafe location to stop. Thetelepresence software application 601 may display an alert dialog box1130, noting that the selected robot destination is an unsafe locationto stop. In the example shown, the alert dialog box 1130 indicates thatthe user selected a ramp 1122 for the robot destination 618 and offersan alternative robot destination 619 just before the ramp 1122. Stoppingthe robot 100 on the ramp 1122 may be hazardous for people near therobot 100 and for the robot 100 itself, if the robot 100 tips or rollsdown the ramp 1122. By determining that the robot destination 618 is onthe ramp 1122, the telepresence software application 601 can eitherprohibit such a robot destination 618 and/or offer a safe alternativedestination 619, in this case before the ramp 1122.

Referring to FIG. 11D, when the user selects an actual obstacle 1120 b,the telepresence software application 601 may display an alert dialogbox 1130, noting that the selected robot destination 618 is outside ofthe navigable area 616 or an obstacle 1120. In the example shown, thealert dialog box 1130 indicates that the user selected an obstacle 1120b for the robot destination 618 and offers an alternative robotdestination 619 just before the obstacle 1120 b.

Referring to FIG. 12, in some implementations, the remote navigationview 612 b displayed by the user interface 605 of the telepresencesoftware application 601 allows a user to specify a robot path 652 to aselected robot destination 618 in the navigable area 616. The user mayspecify the robot path 652 using a variety of input devices. Forexample, on a touch screen display, the user may drag his/her finger ora stylus from the robot icon 650, denoting the current robot position,to the robot destination 618. In additional examples, the user may dragthe robot icon 650 (e.g., using a mouse or touch gesture) along theprescribed robot path 652 to the robot destination 618. In the exampleshown, the user may select a set path button 1202 on the user interface605 allowing the user to indicate that a gesture performed within thenavigable area 616 should be interpreted as the robot path 652. The usermay trace the robot path 652 within the remote video feed window 610.Similarly, the user may select the robot path 652 on the plan view map810 displayed as the two-dimensional map 622 a in the map window 620.After setting the robot path 652, the user may press a go button 1204 toset the robot 100 into motion. Similarly, a stop button 1208 may be usedto stop the motion of the robot 100. A clear path button 1206 may removeor clear the set robot path 652 from the remote navigation view 612 b.

A display window may include a fly-out icon panel that is revealed by amouse-over. For example, the icon panel may “fly” out from the top leftof the window. The icon panel may allow a user to select a manual drive,a click-to-drive, and a head motion icon. In one embodiment, the a usermay toggle through the icons using the space bar. Manual drive may allowa user to click-to-destination and/or click-and-drag a path. Afterdrawing a path on the map, the user may right-click and select “SavePath” from the popup menu. They can give a name to the path. Later theuser may “Load Path”, and the robot will navigate to the starting pointof the path, and then navigate along the prescribed path to thedestination. The path may be stored as a tag data structure, includingtag coordinates and tag information. The tag path include multiple stopsalong the way. When drawing the path, the user may indicate way pointsalong the path. In some embodiments, the way points may be representedby tag annotations that include stop signs. Later, when traversing thepath, upon reaching a way point, the robot may flashes the “stop” buttonand the path may become lighter and/or translucent. At this point theuser may perform a consult and do local driving, then click “go” toresume the path. Accordingly, a physician can save a path for hisevening rounds, hitting all the rooms and stations in a preferred orderand with a pre-planned route.

In head mode, a user may draw a box or outline on a portion of a videofeed in order to center the head (upper portion) of the robot on thecenter of the box or an object within the box. Additionally, a user mayclick a location in order to change the heading of the head (upperportion) of the robot and/or the entire robot. Various button andperipheral control toggles may be used to independently control the base(lower portion) and head (upper portion) of the robot. For example,holding the shift key while in the head mode may make the curser a handicon on the screen and allow the user to grab and drag the viewpoint ofthe head.

In some embodiments, a star icon may be used to control the navigationof the robot. The star icon may be displayed in any of the various viewsand may be selectively moved by the user to change the direction and/orvelocity of the robot. Alternative icons in addition to a star icon maybe used.

Returning to FIG. 12, a virtual joystick window 680 may provide anotherinput device for specifying the desired path 652 or to manually controlof the robot 100. The virtual joystick window 680 may display a robotorientation indicator 682 and a navigation vector 684. A user maycontrol the direction and speed of the robot 100 using the navigationvector 684. A virtual joystick may facilitate control of the robot 100by the user of a device that may not typically have a mouse orconventional joystick, such as a tablet computer.

A “stitched” video image may be displayed in the virtual joystick window680. The “stitch” video and image may be generated using a live downwardpointing camera 320, 450 on the front of the robot 100 and using a livedownward pointing camera on the rear of the robot 100. A user may grab(e.g., using a mouse or touch gesture) and drag on robot motionindicator 686 in order to specify a direction of robot movement and adrive velocity. Driving the robot 100 from the virtual joystick window680 has advantages over using the remote video feed window 610 formouse-based or virtual joystick driving. Specifically, this view mayreduce lens distortion, lack of depth information, and perception issuesbased on the rotation of the robot head 160 that a user may experiencewhile driving the robot 100 using the video feed displayed in remotevideo feed window 610.

In addition to allowing the user to specify a desired path within theremote video feed window 610, the user may specify the robot path 652 onthe map 622 displayed in the map window 620. Specifying the robot path652 in the plan view map window 620 may allow the robot 100 to navigateover longer distances, and thus may free the user to perform other taskswhile the robot 100 is in transit. Various controls may also be providedin order to manipulate the zoom and displayed area of the map 622 shownin the map window 620. A desired zoom may be specified using slider1210, and a desired area may be displayed using an area pan control1212.

Accordingly, a non-technical user may be able to navigate from onelocation to another using any combination of the various navigationmethods and controls. For example, in long-distance travel, a user mayclick a destination on a plan view map and the robot may autonomouslynavigate to the selected location. In mid-range travel, the user mayselect a destination in a video window of a location within the field ofview of the robot. In close range travel, the user may manually controlthe robot's navigation path, rotations, head movements, and the likeusing a mouse, joystick, virtual joystick, or meta-joystick.

FIG. 13 illustrates an exemplary user interface 605 of the telepresencesoftware application 601 having a maximized remote video feed window 610displaying a remote navigation view 612 b that accepts hyper-tags 1310and/or context sensitive commands selectable by the user. The userinterface 605 includes the local video window 630 and the plan view mapwindow 620 overlaid on the remote video feed window 610. In the exampleshown, the plan view map window 620 displays a three-dimensional(three-dimensional) map 622 c. The three-dimensional map 622 c may beutilized by the user to cause the robot 100 to navigatesemi-autonomously to a selected robot destination 618 on thethree-dimensional map 612 c. In some implementations, a virtualthree-dimensional grid 1302 is displayed in the remote navigation view612 b. Using a determined distortion between the plan view map 810 andthe robot map 820, the telepresence software application 601 candetermine the location of the floor surface 5 in the live video feed tooverlay the three-dimensional map 622 c. The user may select a gridsquare 1304 as a robot destination 618 on the virtual grid 1302 in orderto cause the robot 100 to autonomously navigate to the selected gridsquare 1304. The virtual three-dimensional grid 1302 may allow forimproved precision in the positioning of the robot 100.

In the example shown, a variety of hyper-tags (tags) 1310 providingcontext-sensitive actions are displayed and made available to a user.The context-sensitive actions include an approach command 1312 and afollow command 1314. These context-sensitive actions may be generatedupon the identification of a person 1330 within the field of view 322,442, 452 of the robot 100. The user may invoke the approach command 1312in order to position the robot 100 in front of the person 1330. Theapproach command 1312 may cause the execution of an approach behavior512 a (FIG. 5) by the robot behavior system 510 a, whereby the robot 100identifies the person 1330 using its sensor system 400 (e.g., usingfacial recognition) and drives to face the identified person 1330. Theuser may invoke the follow command 1314 to drive the robot 100 behindthe person 1330 and follow at a three-feet distance. The follow command1314 may cause the execution of a person follow behavior 512 b (FIG. 5)by the robot behavior system 510 a, whereby the robot 100 identifies theperson 1330 using its sensor system 400 (e.g., using facial recognition)and drives to follow the identified person 1330. In some examples, therobot 100 may detect individuals within its field of view 322, 442, 452using a facial recognition routine. A label 1340 may be displayed thatidentifies the individual. For example, the information may includename, title, occupation, address, business address, email address,web-page address, user notes, etc.

The telepresence software application 601 may determine a distortionbetween the displayed two-dimensional map 622 a and the first-personvideo feed captured by the robot camera 320. Determining such adistortion may include determining a coordinate transformation betweenthe two-dimensional map and the three-dimensional “map.” When the userplaces a tag 662 and/or hyper-tag (which may comprise a tag) 1310 eitheron the remote view 612 of the remote video feed window 610 or on thetwo-dimensional map 622 a of the map window 620, the telepresencesoftware application 601 may apply the distortion to tag map coordinatesassociated with the tag 662 and/or the hyper-tag 1310 to determinecorresponding video coordinates or plan view map coordinates,respectively, and overlay a tag annotation associated with the tag 662or hyper-tag 1310 on the displayed remote view 612 (i.e., first-personvideo feed) or the map 622, respectively, using the determined video ormap view coordinates. In various embodiments, the three-dimensionalrendition of the tag annotation may be dynamically re-rendered based onthe current position of the remote telepresence robot and a perspectiveof the tag relative to the video feed. Accordingly, as the location ofthe robot and/or the perspective of the live video feed changes, such aswhen the head (upper portion) of the robot pans or tilts, the tagannotation may be dynamically re-rendered. For example, a tag annotationcorresponding to a ramp may be overlaid in the video feed with respectto the floor. Similarly, a tag annotation associated with an object on awall may be overlaid with respect to the object or the wall.

As described herein, the tag may include tag information comprising arobot action modifier. The tag may be interpreted by a robot operator, alocal terminal, or the remote telepresence robot and cause the robot toexecute a predetermined action. For example, the robot action modifiermay direct a robot to not enter a specific area, to travel slow througha certain area, to travel fast through a certain area, to use extracaution, and/or to perform other actions. Tags in general may includeany of a wide variety of information, such as an availability of awireless communication signal, a speed the remote telepresence robotshould travel, a location of a point of interest, a location of aperson, a location of a docking station, a location of a rest area, alocation of a glass wall, a location of a ramp, a location of an object,an optimal route to navigate a tight area, an optimal rout to navigate acongested area, and an action a remote telepresence robot shouldexecute. The tag may be created by a user, automatically by a terminal,automatically by a robot, and/or in response to historical datacollected by the terminal and/or robot.

The robot may include a tag identification system configured to identifytags having tag coordinates encountered along a navigation path. A robotmay “encounter” a tag when the tag coordinates are within the localperceptual space of the robot and/or the tag coordinates are relevant toan objective, planned navigation path, or the planning of a navigationpath. Accordingly, a tag identification system may “encounter” a tagalong a navigation path, even if the robot is not yet in proximityand/or may never be in proximity to the tag coordinates of the tag.

A robot and/or remote terminal determining a navigation path for a robotmay take into account tags or potential tags that could influence thenavigation path. Accordingly, the tag identification system may be usedto identify tags having tag coordinates projected to be along potentialnavigation paths during the determination of a navigation path. Forinstance, several potential navigation paths may be used to reach adesired destination and the selection of which navigation path will beused may depend on the tags relevant to each of the potential navigationpaths. A robot selecting between multiple potential navigation paths mayidentify relevant tags in order to determine which navigation path wouldprovide the best wireless connectivity. Other factors, such as ramps,elevators, distance, congestion, objects,

In the exemplary user interface 605 shown, the dashboard window 640provides a battery charge status, a wireless signal strength indicator,and a robot outline having portions that may light up when service isrequired. An options window 690 allows the user to disconnect or dockthe robot with a docking station and set software and/or robot options.

Referring to FIG. 14, in some implementations, while executing theperson follow behavior 512 b, the robot 100 may detect, track, andfollow a person 1330. Since the robot 100 can pan and tilt the head 160using the neck 150, the robot 100 can orient the secondthree-dimensional image sensor 450 b to maintain a corresponding fieldof view 452 on the person 1330. Moreover, since the head 160 can moverelatively more quickly than the base 120 (e.g., using the drive system200), the head 160 (and the associated second three-dimensional imagesensor 450 b) can track the person 1330 more quickly than by turning therobot 100 in place. The robot 100 can drive toward the person 1330 tokeep the person 1330 within a threshold follow distance range D_(F)(e.g., corresponding to a sensor field of view). In some examples, therobot 100 turns to face forward toward the person/user 1330 whiletracking the person 1330. The robot 100 may use velocity commands and/orwaypoint commands to follow the person 1330.

Additional details and features concerning person recognition and personfollowing can be found in PCT application serial number PCT/US11/35488,filed on May 6, 2011, which is hereby incorporated by reference in itsentirety.

FIGS. 15A and 15B illustrate alternative three-dimensional maps 622 cand two-dimensional maps 622 a displayable in the plan view map window620 that include hyper-tags 1310 associated with various information andthat may be used to cause the robot to navigate autonomously to aparticular destination. The hyper-tags 1310 may include informationabout various locations or information related to patients. A user mayadd labels 1502 or mark-ups 1504, such as personal notes, shared notes,sketches, drawings, etc. A robot location 1510 may also be identified. Auser may specify a robot destination 1512, such as a nurse's station.The robot 100 may navigate autonomously to the specified robotdestination 1512.

The telepresence software application 601 may display information on theremote video view 612 and/or a map 622, indicating physical areas ofinterest. For example, a small arrow with an attached bubble reading“Pharma” may indicate the location of the pharmacy room. Such a bubblecould comprise a tag. For example, the tag could include tag coordinatesindicating where the word “Pharma” should be displayed; tag information,such as relevant information related to the pharmacy; and a tagannotation, such as a graphical representation in two-dimensional and/orthree-dimensional of the word “Pharma.” In some examples, the user candetermine what information is available about nearby rooms by placingthe mouse over or gesturing over that area, causing the display of anycorresponding available information. With this information, the user canquickly choose to go to a destination (e.g., the pharmacy room) byselecting a robot destination 618 in the remote navigation view 612 b(FIG. 12).

For example, according to one example a robot or remote terminal couldretrieve tag coordinates that correspond to tags associated with a robotmap. Using the robot position, tags that are in close proximity to therobot may be identified. Tags within the field of view of the robot maybe identified using the orientation of the robot's head (upper portion).The robot and/or remote terminal could then calculate a set ofcoordinates for all of the pixels on the video screen and render a tagannotation associated with each tag within the line of sight based onthe position of the robot and the perspective provided by the robot'scurrent head orientation (pan and/or tilt). According to someembodiments, Denavit-Hartenberg parameters (DH parameters) may be usedas a standard coordinate system for spatial linkages between the videofeed and the map plan view.

Referring again to FIG. 8E, the tagging view 660 of the user interface605 allows the user to place tags 662 on a plan view map 810 todesignate locations of interest and/or mark the plan view map 810 withinformation, such as obstacles, preferred robot travel routes, etc.Referring also to FIG. 13, the user may place hyper-tags 1310 on theremote navigation view 612 b to mark locations with context-sensitiveinformation. The map data source 1620 may store the tagging andhyper-tag information (e.g., locations, tag identifiers, tag content)along with layout map and/or robot map information. As used herein,hyper-tags may be embodied as tags and use a similar data structure totags, as described herein.

In addition or alternatively to allowing the user to place tags 662 andhyper-tags 1310 in the user interface 605, the user may enteruser-specific hyper-tags 1310 during operation of the robot 100. Theuser may invoke a command that allows for the insertion of a hyper-tag1310 at a current robot location. Another command may allow for theremoval of the hyper-tag 1310. Further, other users (e.g., nurses) maybe allowed to add hyper-tags 1310 that may be shown to a user of therobot 100. A “nurse map application” may display a top-down map or atagging view 660 that allows placement of temporary hyper-tags 1310, forexample, to identify rooms of interest to a doctor who may soon belogging in. Moreover, some hyper-tags 1310 may be user-specific and/ortime-specific. For example, a stroke patient in a room may be showingsigns of deterioration. The nurse can call up the “nurse mapapplication,” find that room on the map and enter a hyper-tag 1310. Thenurse may fill in a hyper-tag as follows: hyper-tag_name=“Stroke patientdeteriorating,” user_specific=“Dr. Reynolds,” duration=1 hour. Thus, ifDr. Reynolds logs in within the next hour, he would see a hyper-tag 1310associated with the patient's room on the map additionally indicating“Stroke patient deteriorating.” On approaching that wing, he may alsosee a hyper-tag 1310 pop up pointing to that room in the video stream,labeled “Stroke patient deteriorating.” No other doctor would see thoselabels, and Dr. Reynolds would only see them during that first hour.

A doctor may also set up temporary bookmark and reminder hyper-tags1310, directly at a local or remote station interface 606, 608, toassist with his/her work plan. In some examples, the doctor may assignnumbers to several patient rooms at the start of the session. Thenduring the session, he/she may see the numbers on the displayed map 622and in popup hyper-tags 1310 to remind him/her of the order in which tovisit the patients 614. The doctor may add notes which can be viewedthrough the remainder of the session or upon next returning, forexample, “come back at end of session” on one patient, or “writeprescription” or “check in again at 4 pm.”

Additionally, “smart” hyper-tags 1310 may be displayed automatically.For example, a nurse may enter photos of incoming patients 614 into adatabase (e.g., stored locally and/or on cloud storage 722)cross-referenced with their electronic medical record. The telepresencesoftware application 601 may execute a face recognition algorithm on avideo stream captured by the robot camera 320 to identify the patients614, 1330, which can be cross-referenced to the database. Uponrecognition of a patient's face, the telepresence software application601 may automatically pull up and display the patient's electronicmedical record.

Referring again to FIG. 14, in some implementations, each patient 614,1330 receives a radio frequency identification (RFID) chip 497, such ason a wristband. The robot 100 may have an RFID reader 498 incommunication with the controller 500 as part of its sensor system 400to recognize nearby patients via the RFID chip. The telepresencesoftware application 601 may display a corresponding hyper-tag when thepatient comes within RFID range (e.g., six feet) of the robot 100. Thehyper-tag 1310 may appear to be floating in the air, since RFID is notdirection-specific. An alternative hybrid approach may use computervision techniques to identify the existence of a patient 614, 1330 inthe field of view 322 of the robot 100 by identifying a human face, andthen assuming that the RFID match belongs to that patient and localizingthe hyper-tag 1310 on the patient 614, 1330.

Referring to FIGS. 16A-16D, in some implementations, a robot system 1600includes one or more telepresence robots 100 in communication with abridge 602, which communicates with a local robot endpoint server 604 aand a remote endpoint server 604 b (e.g., such as the cloud computingservice 720 (FIG. 7)). The local robot endpoint server 604 acommunicates with a local technician computing device 606 and the remoteendpoint server 604 b communicates with a remote operator computingdevice 608. The robot system 1600 also includes one or more data sources1610 for storing sensor data received from the robot sensor system 400and/or user interaction data, such as information obtained from the userthrough the web pad 310 and/or the user interface 605. In the exampleshown, the robot system 1600 includes at least one robot sensor datasource 1610 a for storing sensor data and at least one head data source1610 b for storing user interaction data. The data sources 1610 mayreside on the robot 100, cloud storage 722 (FIG. 7), the local robotendpoint server 604 a and/or the remote endpoint server 604 b.

A map data source 1620, such as a database stored on the robot 100,cloud storage 722, the local robot endpoint server 604 a and/or theremote endpoint server 604 b, can store information for the plan viewmap 810, the robot map 820, tag 662 information, and/or hyper-tag 1310information. The map data source 1620 may be a single database orcombination of data sources 1610, such as the robot sensor data source1610 a and the head data source 1610 b. The telepresence softwareapplication 601 and/or the robot 100 (e.g., the controller 500) mayaccess the map data source 1620 to execute real-time or off-lineconcordance processing, provide user interface feedback, performnavigation routines, render maps 622, etc.

In some implementations, the control system 510 executing on thecontroller 500 of the robot 100 accesses one or more of the data sources1610, such as the robot sensor data source 1610 a, the head data source1610 b, and/or the map data source 1620 to issue events recognizable bythe behavior system 510 a. In response to raised events, the behaviorsystem 510 a may execute one or more behaviors 512 that affect theselection of a command executed by the resource control arbiter 560 onthe robot resources 530 (FIG. 5). In the example shown in FIG. 16C, therobot control system 510 communicates with the map data source 1620 toaccess a concordance matrix/database, which may store concordanceprocess information, such as real-time sensor/flag data 1622 a, operatorcommands 1622 b, local perceptual space data 1622 c (e.g., volumetricpoint cloud data received from a three-dimensional image sensor 450),occupancy bitmap data 1622 d (e.g., the robot map 820), floor plan data1622 e (e.g., the plan view map 810), and an end-user tag table 1622 f(e.g., storing x, y, z coordinates and tag fields), and/or a robotbehavior tag table 1622 g (e.g., storing x, y, z coordinates and tagfields). Referring also to FIG. 5, behaviors 512 of the behavior system510 a may evaluate possible outcomes of robot actions based on raisedevents, such as sensor events from the sensor system 400 and tag events(e.g., which may mimic a sensor event) raised by placed tags 662, 1310stored in the tag tables 1622 f, 1622 g. Accordingly, the actionselection engine 580 may select a feasible robot action having the bestoutcome based on behavior evaluations. As a result, the robot 100 mayautonomously operate in a manner that takes into account the tags 662,1310 received by the telepresence software application 601.

Referring again to the ramp example shown in FIG. 11C, when the robot100 approaches a ramp 1122, the robot control system 510 may perceivethe ramp 1122 as an obstacle 1120, based on sensor signals received fromthe sensor system 400. In order to discern between a perceived obstacle1120 a and an actual obstacle 1120 b, the control system 510 may need toaccess a common database, such as the map data source 1620, storingrobot data and user data. Using the map data source 1620, the controlsystem 510 can determine that the detected ramp 1122 is a perceivedobstacle 1120 a, rather than an actual obstacle 1220 b. Moreover, thecontrol system 510 may communicate with the telepresence softwareapplication 601 to receive a user input as to whether the user perceivesthe ramp 1122 as an actual obstacle 1120 b and/or to receive analternative robot path 652 and/or an alternative robot destination 619.The telepresence software application 601 can use the map data source1620 to resolve distortions between two-dimensional maps 622 a, 810 andthree-dimensional maps 622 c, between live video feeds in the remoteview 612 and two-dimensional and/or three-dimensional maps 622 a, 622 cto provide hybrid maps 622 b (FIG. 9B). Moreover, the telepresencesoftware application 601 can use the map data source 1620 to render thelook-ahead view 612 a in the plan view map window 620 (FIG. 10C).

Referring again to FIG. 12, in additional implementations, when the userselects a robot path 652 on one of the two-dimensional map 622 a, hybridmap 622 b, three-dimensional map 622 c, and the remote view 610, thetelepresence software application 601 can use the map data source 1620to resolve distortions between any of the maps 622 and the remote view612 and apply the distortion to the selected robot path to determine acorresponding sequence of robot path map coordinates on the robot map820 for use by the robot controller 500 when executing a drive commandto the destination 618. Moreover, the telepresence software application601 may apply the determined distortion(s) to resolve correspondingsequences of robot path coordinates for displaying the robot path 652 onany of the maps 622 and the remote view 612. The map data source 1620may store the determined distortions and/or the sequences of robot pathcoordinates for displaying the robot path 652 on any of the maps 622 andthe remote view 612.

Accordingly, it should be broadly understood that the term “distortion”as used herein relates to resolving spatial coordinate errors,differences of transformation from one coordinate system to another,including between coordinates systems of different dimensions. Forexample, a robot and/or remote terminal may determine a distortionbetween a two-dimensional plan view map and a two-dimensional mapgenerated, at least in part, by a robot, such as those generated usingvarious robot sensor or laser scans. Additionally, a robot and/or remoteterminal may determine a distortion between a three-dimensional map orvideo feed and a two-dimensional plan view map. Moreover, determining adistortion may relate to transforming coordinates between first personviews, third person views, plan view map views, hybrid map views, and/orbetween any two different coordinate systems or perspectives within thesame coordinate system.

Referring again to FIG. 13, in some implementations, when the userplaces a tag 662 or hyper-tag 1310 on the plan view map 810 displayed asthe map 622 in the map window, the telepresence software application 601determines a user selected location on an electronic display displayingthe map 622 and overlays an annotation associated with the tag 662, 1310on the map 622. The telepresence software application 601 may alsodetermine a distortion between the plan view map 810 and the remote view610 (i.e., first-person video captured by the robot camera 320) andapply the distortion to coordinates of the tag 662, 1310 on the planview map 810 to determine corresponding video coordinates of the removeview 610. A tag annotation associate with the tag 662, 1310 and storedby the map data source 1620 can be displayed by the telepresencesoftware application 601 on the remote view 610 using the determinedvideo coordinates.

Referring to FIG. 17, in some implementations, the user interface 605provides an augmented overlay 1710, for example, in the remote videofeed window 610 and/or the map window 620, that allows the user tovisualize a position of the robot head 160 with respect to the robotbase 120. The augmented overlay 1710 may allow the user to appreciate acurrent field of view 322, 442, 452, denoted by an arc 1720 in theexample shown, of the robot sensor system 400 relative to a full 360degrees field of view. This allows the user to make selections forrotation (e.g., of the head 160 and/or base 120) which are outside thecurrent field of view 322, 442, 452.

The user may click within the zone defined by first and second rings1722 and 1724 in order to rotate a virtual head 1728 to that point. Asthe user rotates the virtual had 1728, the robot head 160 may move inreal time with the telepresence software application 601 updating thelive video feed from the robot camera 320 displayed in the remote view612 of the remote video feed window 610 in real time as well. In theexample shown, the augmented overlay 1710 has a virtual base 1726corresponding to the robot base 120 and a virtual head 1728corresponding to the robot head 160 arranged at an angle/orientationwith respect to the virtual base 1726 that corresponds to a current poseof the robot 100. In some examples, one of the virtual base 1726 and thevirtual head 1728 is shown static while the other is free to moverelative to the static one.

If the user clicks within the zone defined by the first and second rings1722 and 1724 to rotate the head 1728 outside of the current field ofview 1720, the robot 100 may rotate the head 160 and/or the base 120 inorder to accomplish the user's command. In some examples, after rotatingthe head 160 according to the user command, the base 120 may rotate andthe head 160 may then move to a center position. Such changes inposition may be problematic if the user then attempts to reposition therobot 100 based on the previous rotation. To mitigate this, certainimplementations may employ a system to reduce the requirement of baserotations in order to accommodate head rotations. For example, a countermay be initiated when the virtual head 1728 is turned to an angle. Ifthe robot head 160 remains at that angle for a specified interval, thesystem may slowly rotate the base 120 in order to center the head 160with respect to the base 120 while simultaneously rotating the head 160in the opposite direction at the same speed. This keeps the currentsubject in view, while also ensuring that the head 160 and the base 120are now in alignment and the forward frame of reference is dictated bywhere the user is looking. Further, if the user wishes to continuelooking farther in that direction, the full panning range of motion ofthe head 160 is available.

FIG. 18 illustrates an exemplary sequence of robot events for respondingto a user command, for example, in the telepresence software application601. In an initial state, the robot 100 may receive a drive command tomove from one location to another. The command may beoperator-initiated, behavior-initiated (e.g., of a behavior executing ona control system of the controller), or planner-initiated (e.g., apre-planned task or routine). In this example, the command includes anew heading for moving in an opposite direction to a new destination. Inresponse to the command, the robot 100 may turn its head 160 (left orright) toward a panning limit. After reaching the panning limit, therobot 100 may rotate the base 120 (e.g., holonomically in place), toallow movement of the head 160 in order for the head to turn toward thenew heading. The term “panning limit” may refer to a point when theupper portion of the robot cannot physically rotate anymore with respectto the lower portion of the robot, a point where the upper portion ismisaligned with respect to the lower portion a predefined number ofrotational degrees, and/or the term panning limit” may be a function ofthe number of degrees the upper portion is misaligned with respect tothe lower portion and the length of time the upper portion has beenmisaligned with respect to the lower portion.

In some examples, the robot 100 continues to rotate the base 120, sothat the forward drive direction F coincides with the new heading, thusproviding the head 160 relatively equal left/right panning ability. Asthe robot 100 rotates the base, it may simultaneously rotate the head160 so as to face the heading and optionally so that a field of view322, 452 of a sensor 320, 450, 450 b on the head 160 can point along thenew heading. In some implementations, the robot 100 turns the base 120and the head 160 together, so as to allow the head 160 to face the newheading relatively quicker. If the base 120 over-rotates, the head 160can counter-pan to recover alignment.

FIG. 19 illustrates an exemplary remote view 612 where the telepresencesoftware application 601 overlays a screen indicator 1910 on the remotevideo feed received from the robot 100. The screen indicator 1910 may bedisplayed near a mouse cursor 1912 and may represent a current headrange of motion. As the user moves the mouse cursor 1912 toward the leftor right side of the remote video view 612 (with the probable intentionof clicking to move the head to point there), the on screen indicator1910 may be displayed above the cursor 1912 to indicate how much headmotion remains in that direction (e.g., how much remaining range ofmotion of the head 160 is available).

A highlight box 1920 may highlight an area of interest within the remotevideo view 612. The user may create the highlight box 1920 around anarea of interest on a portion of the remote video view 612, for example,by dragging and dropping a box onto the screen and/or by clicking anddragging open a box around the area of interest. In response, thetelepresence software application 601 may cause the robot head 160 tomove to center on the highlight box 1920. Moreover, the camera 320 mayzoom in to match the dimensions of the highlight box 1920.

Referring to FIGS. 20A-20B, in some implementations, if the robot 100unexpectedly loses communication connectivity (e.g., a loss of thewireless signal), the robot 100 may stop or continue driving to itsdestination. As a telepresence robot moves throughout an environment,communication may be disrupted, for example, as the robot 100transitions between various wireless access points and/or encountersdisruptions in the data transmission as a result of poor signalstrength. By continuing to navigate autonomously, communication may berestored by the time the robot arrives at a desired destination.

When the robot 100 experiences a loss of communication connectivity, therobot 100 may refer to a last trusted localization/pose (e.g., storedlocally by the controller 500) and/or a current determinedlocalization/pose (e.g., based on the robot sensor system 400) tocontinue navigating to the destination. If the robot path is a plannedpath, the robot 100 may resume the planned path to the destination. Onthe other hand, if the user was teleoperating the robot 100 to thedestination, the robot 100 may follow a planned path to a nearest/lasttrusted location having communication connectivity (e.g., radiofrequencyand/or wireless). Alternatively, the robot 100 may drive along ashortest path (i.e., a new path) to a nearest/last trusted locationhaving communication connectivity.

After reaching the nearest/last trusted location, the robot 100 maydetermine if communication connectivity has been reestablished and ifso, whether the destination has been reached. If communicationconnectivity was not reestablished, the robot 100 may synchronize itsvideo recordings (and any other sensor data of the sensor system 400) tomove to a next trusted location stored by the controller 500. Moreover,if the destination has not been reached, but communication connectivitywas reestablished, the controller 500 may execute safe harborcountermeasures, which may entail continuing recording sensor data anddisplaying a new robot-side user interface (e.g., on the web pad 310)indicating that the session was terminated due to loss of communicationconnectivity. The robot 100 may improve its connectivity recoverypercentage by reassessing and/or executing path planning to move to thelast trusted location (using ODOA). The robot 100 may also move itsantennas 490 a, 490 b (FIGS. 2 and 4C) to possibly gain bettercommunication reception. The robot 100 may use a mobile ad-hoc network(MANET), which is a self-configuring infrastructureless network ofmobile devices connected by wireless links.

In some examples, the robot 100 may improve the integrity/accuracy ofthe robot map 820 by marking the location of lost communications and anylocation of reestablished communications. The robot 100 may use waypointnavigation to move to an area having known connectivity (e.g., a WLANwarm zone or high signal zone). Waypoints are sets of coordinates thatidentify a point in physical space. The robot 100 may use waypointsestablished on the robot map 820 to maneuver to the destination.

Additional safe harbor countermeasures may include planning a path to anearest lowest traffic area or moving to a nearest charging/dockingstation based on the robot map 820. In some examples, the robot 100 maymove toward another nearest robot 100, which may use multiple antennas490 a, 490 b for multiple-input and multiple-output (MIMO) to act as aWi-Fi bridge 602.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed application specific integrated circuits (ASICs),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

Implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer-readable medium forexecution by, or to control the operation of, data processing apparatus.The computer-readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “data processing apparatus” encompassesall apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multipleprocessors or computers. The apparatus can include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a standalone program or as a module,a component, a subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., a field programmable gate array (FPGA) or an ASIC.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of a digital computer. Generally,a processor will receive instructions and data from a read only memoryor a random access memory or both. The essential elements of a computerare a processor for performing instructions and one or more memorydevices for storing instructions and data. Generally, a computer willalso include, or be operatively coupled to receive data from or transferdata to, or both, one or more mass storage devices for storing data,e.g., magnetic disks, magneto optical disks, or optical disks. However,a computer need not have such devices. Moreover, a computer can beembedded in another device, e.g., a mobile telephone, a personal digitalassistant (PDA), a mobile audio player, a Global Positioning System(GPS) receiver, to name just a few. Computer-readable media suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back endcomponent, e.g., as a data server; that includes a middleware component,e.g., an application server; or that includes a front end component,e.g., a client computer having a graphical user interface or a webbrowser through which a user can interact with an implementation of thesubject matter described is this specification, or any combination ofone or more such back end, middleware, or front end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of a client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what may beclaimed, but rather as descriptions of features specific to particularimplementations of the invention. Certain features that are described inthis specification in the context of separate implementations can alsobe implemented in combination in a single implementation. Conversely,various features that are described in the context of a singleimplementation can also be implemented in multiple implementationsseparately or in any suitable sub-combination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a sub-combination or variation ofa sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multi-tasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims. Forexample, the actions recited in the claims can be performed in adifferent order and still achieve desirable results.

1. A telepresence robot, comprising: an upper portion; a lower portionrotatably connected to the upper portion; a drive system configured tomove the telepresence robot according to drive instructions; a controlsystem in communication with the drive system, the control systemconfigured to generate drive instructions to cause the drive system tomove the telepresence robot; and a rotation system configured to rotatethe robot from a first heading to a second heading by rotating the upperportion and the lower portion independently.
 2. The telepresence robotof claim 1, wherein the rotation system is configured to rotate therobot toward a second heading by: rotating the upper portion of therobot toward the second heading; detecting that the upper portion of therobot has reached a panning limit of the upper portion of the robotrelative to the lower portion of the robot; beginning to rotate thelower portion of the robot toward the second heading at the panninglimit of the upper portion of the robot; detecting that the upperportion of the robot has reached the second heading; and continuing torotate the lower portion of the robot toward the second heading whilesimultaneously counter-rotating the upper portion of the robot, suchthat the upper portion of the robot maintains the second heading.
 3. Thetelepresence robot of claim 2, wherein the panning limit is reached whenthe upper portion cannot physically rotate anymore with respect to thelower portion of the robot.
 4. The telepresence robot of claim 2,wherein the panning limit is reached when the upper portion ismisaligned with respect to the lower portion a predefined number ofrotational degrees.
 5. The telepresence robot of claim 2, wherein thepanning limit is a function of the number of degrees the upper portionis misaligned with respect to the lower portion and the length of timethe upper portion has been misaligned with respect to the lower portion.6. The telepresence robot of claim 1, wherein the rotation system isconfigured to rotate the robot toward a second heading by: rotating theupper portion of the robot toward the second heading at a firstrotational velocity; rotating the lower portion of the robot toward thesecond heading at a second rotation velocity; detecting that the upperportion of the robot has reached the second heading; and continuing torotate the lower portion of the robot toward the second heading whilesimultaneously counter-rotating the upper portion of the robot, suchthat the upper portion of the robot maintains the second heading.
 7. Thetelepresence robot of claim 1, further comprising: an imaging system incommunication with the control system; and a positioning system incommunication with the control system configured to provide a currentposition of the robot relative to a plan view map and a currentalignment of the upper portion with respect to the plan view map, andwherein the control system is configured to transmit a video feed fromthe imaging system, the current position of the robot, and the currentalignment of the upper portion to a remote terminal, such that theremote terminal can: determine a distortion between the plan view mapand the video feed received from the imaging system of the remotetelepresence robot; apply the distortion to a tag having coordinatesassociated with the plan view map in order to determine correspondingvideo coordinates and perspective data describing the location andperspective of the tag relative to the video feed; and display athree-dimensional rendition of the tag overlaid on the video feed usingthe video coordinates.
 8. The telepresence robot of claim 1, furthercomprising: an imaging system in communication with the control system;and a positioning system in communication with the control systemconfigured to provide a current position of the robot relative to a planview map and a current alignment of the upper portion with respect tothe plan view map, and wherein the control system is configured to:transmit a video feed from the imaging system, the current position ofthe robot, and the current alignment of the upper portion to a remoteterminal; determine a distortion between the plan view map and the videofeed received from the imaging system of the remote telepresence robot;and apply the distortion to a tag having coordinates associated with theplan view map in order to determine corresponding video coordinates andperspective data describing the location and perspective of the tagrelative to the video feed, such that the remote terminal can display athree-dimensional rendition of the tag overlaid on the video feed usingthe video coordinates.
 9. A telepresence robot system local terminalcomprising: an electronic display; a processor in communication with theelectronic display; and a memory in communication with the processor,the memory comprising instructions executable by the processorconfigured to cause the processor to: retrieve at least a portion of aplan view map representative of robot-navigable areas of a robotoperating surface; retrieve at least one of a plurality of tags, each ofthe plurality of tags comprising tag coordinates describing the relativelocation of the tag and tag information; receive a video feed from animaging system of a remote telepresence robot; receive positioninginformation associated with a current position of the remotetelepresence robot; display the video feed from the imaging system ofthe remote telepresence robot; display a rendition of the taginformation of the at least one tag on the video feed using the tagcoordinates; and transmit a command to the remote telepresence robot.10. The telepresence robot system of claim 9, wherein the instructionsexecutable by the processor are further configured to cause theprocessor to: determine a distortion between the plan view map and thevideo feed received from the imaging system of the remote telepresencerobot; apply the distortion to the tag coordinates of the at least onetag to determine corresponding video coordinates and perspective datadescribing a location and perspective of the at least one tag relativeto the video feed; and display a three-dimensional rendition of the taginformation of the at least one tag overlaid on the video feed using thetag video coordinates.
 11. The telepresence robot system of claim 10,wherein the three-dimensional rendition of the tag information isdynamically re-rendered based on the current position of the remotetelepresence robot and the perspective of the at least one tag relativeto the video feed.
 12. The telepresence robot system of claim 10,wherein the three-dimensional rendition of the tag information isoverlaid on the video feed with respect to an object detected in thevideo feed.
 13. The telepresence robot system of claim 12, wherein thethree-dimensional rendition of the tag information is overlaid along awall detected in the video feed.
 14. The telepresence robot system ofclaim 9, wherein the tag information of the at least one tag comprises arobot action modifier, and wherein the robot action modifier isconfigured to provide execution instructions to a control system of thetelepresence robot to execute a first action in response to thetelepresence robot being within a predetermined range of the tagcoordinates of the at least one tag.
 15. The telepresence robot systemof claim 14, wherein the instructions executable by the processor arefurther configured to cause the processor to transmit the executioninstruction to the control system of the telepresence robot when thetelepresence robot is within a predetermined range of the tagcoordinates of the at least one tag.
 16. The telepresence robot systemof claim 14, wherein the robot action modifier further comprisesinstructions regarding one of a time and a location on the plan view mapthat the control system of the telepresence robot should execute thefirst action.
 17. The telepresence robot system of claim 9, wherein theinstructions executable by the processor are further configured to causethe processor to: receive a sequence of coordinates relative to the planview map forming a path along which the remote telepresence robot hastraveled; store the sequence of coordinates forming the path as a pathtag comprising tag coordinates and tag information; retrieve the pathtag when the remote telepresence robot arrives within a predetermineddistance of the tag coordinates; and display a rendition of the taginformation of the path tag on the video feed using the tag coordinates.18. The telepresence robot system of claim 17, wherein the telepresencerobot system local terminal further comprises at least one user inputdevice, and wherein the sequence of coordinates forming the path isprovided by the user input device.
 19. The telepresence robot system ofclaim 17, wherein the sequence of coordinates forming the path isprovided by the remote telepresence robot.
 20. The telepresence robotsystem of claim 9, further comprising a communication system configuredto facilitate communication between the telepresence robot system localterminal and the remote telepresence robot.
 21. The telepresence robotsystem of claim 9, wherein the local terminal further comprises at leastone user input device, and wherein the user input device is configuredto allow a user to provide an indication of a desired destination of theremote telepresence robot on at least one of the plan view map and thevideo feed from the imaging system of the remote telepresence robot, andwherein the command transmitted to the remote telepresence robotcomprises the desired destination.
 22. The telepresence robot system ofclaim 21, wherein the sequence of coordinates forming the robot path isbased at least in part on tagging information associated with the atleast one tag.
 23. The telepresence robot system of claim 21, whereinthe instructions executable by the processor are further configured tocause the processor to: determine a sequence of coordinates relative tothe plan view map to create a robot path between the current position ofthe remote telepresence robot and the desired destination of the remotetelepresence robot, and wherein the command transmitted to the remotetelepresence robot comprises the sequence of coordinates forming therobot path.
 24. The telepresence robot system of claim 23, wherein theinstructions executable by the processor are further configured to causethe processor to display the sequence of coordinates forming the robotpath overlaid on the plan view map.
 25. The telepresence robot system ofclaim 23, wherein the instructions executable by the processor arefurther configured to cause the processor to: determine a distortionbetween the plan view map and the video feed received from the imagingsystem of the remote telepresence robot; apply the distortion to thesequence of coordinates forming the robot path to determinecorresponding video coordinates and perspective data describing alocation and perspective of the sequence of coordinates relative to thevideo feed; and display a three-dimensional rendition of the sequence ofcoordinates forming the robot path overlaid on the video feed.
 26. Thetelepresence robot system of claim 25, wherein the three-dimensionalrendition of the sequence of coordinates forming the robot path isoverlaid on the video feed with respect to a floor detected in the videofeed.
 27. The telepresence robot system of claim 9, wherein theinstructions executable by the processor are further configured to causethe processor to: receive a sequence of coordinates relative to the planview map from a navigation system of the remote telepresence robot, thesequence of coordinates forming a robot path between the currentposition of the remote telepresence robot and a desired destination ofthe remote telepresence robot; and display the sequence of coordinatesforming the robot path overlaid on the plan view map.
 28. Thetelepresence robot system of claim 27, wherein the instructionsexecutable by the processor are further configured to cause theprocessor to: determine a distortion between the plan view map and thevideo feed received from the imaging system of the remote telepresencerobot; apply the distortion to the sequence of coordinates forming therobot path to determine corresponding video coordinates and perspectivedata describing the location and perspective of the sequence ofcoordinates relative to the video feed; and display a three-dimensionalrendition of the sequence of coordinates forming the robot path overlaidon the video feed.
 29. The telepresence robot system of claim 28,wherein the three-dimensional rendition of the sequence of coordinatesforming the robot path is overlaid on the video feed with respect to afloor detected in the video feed.
 30. The telepresence robot system ofclaim 9, wherein the tag information relates to at least one of aposition, a path, and a volume, and wherein the control system isconfigured to execute an action relative to the at least one of theposition, the path, and the volume.
 31. The telepresence robot system ofclaim 9, wherein the instructions executable by the processor arefurther configured to cause the processor to receive coordinates on theplan view map of an obstacle detected by a sensor system of the remotetelepresence robot.
 32. The telepresence robot system of claim 9,wherein the plan view map and the plurality of tags are stored remotely.33. The telepresence robot system of claim 31, wherein the plan view mapand the plurality of tags are stored within the remote telepresencerobot.
 34. The telepresence robot system of claim 9, wherein theinstructions executable by the processor are further configured to causethe processor to: determine a distortion between the plan view map andthe video feed received from the imaging system of the remotetelepresence robot; and generate a hybrid map view comprising a blendedview of the plan view map and the video feed from the imaging system ofthe remote telepresence robot.
 35. The telepresence robot system ofclaim 34, wherein the hybrid map view comprises a three-dimensionalrepresentation of the plan view map overlaid on the video feed.
 36. Thetelepresence robot system of claim 9, wherein the telepresence robotsystem local terminal further comprises at least one user input device,and wherein the instructions executable by the processor are furtherconfigured to cause the processor to: receive a request via the at leastone input device for a rendered look ahead for a virtual location of theremote telepresence robot on the plan view map; determine a distortionbetween the plan view map and the video feed received from the imagingsystem of the remote telepresence robot; generate a virtualthree-dimensional video feed based on a virtual location of the remotetelepresence robot; and display the virtual three-dimensional video feedbased on the virtual location of the remote telepresence robot.
 37. Thetelepresence robot system of claim 9, wherein the tag information of theat least one tag comprises a set of coordinates with respect to the planview map defining a protected region, and wherein the tag information ofthe at least one tag is configured to indicate the presence of theprotected region.
 38. The telepresence robot system of claim 9, whereinthe instructions executable by the processor are further configured tocause the processor to: receive a request to create a new tag; associatetag coordinates describing a relative location of the new tag and taginformation with the new tag; and display a rendition of the taginformation of the new tag on the video feed using the tag coordinates.39. The telepresence robot system of claim 38, wherein the request tocreate the new tag is generated by the remote telepresence robot. 40.The telepresence robot system of claim 38, wherein the request to createthe new tag is automatically generated based on a detected object in thevideo feed.
 41. The telepresence robot system of claim 40, wherein thenew tag is a temporary tag configured to expire once the detected objectis no longer present in the video feed.
 42. The telepresence robotsystem of claim 40, wherein the object is a person and the taginformation of the new tag comprises identification informationassociated with the person.
 43. The telepresence robot system of claim40, wherein the object is a person and the tag information of the newtag comprises potential actions the remote telepresence robot canexecute with respect to the person.
 44. The telepresence robot system ofclaim 38, wherein the request to create the new tag is generated by auser input device in communication with the telepresence robot systemlocal terminal.
 45. The telepresence robot system of claim 40, whereinthe request to create the new tag is made with respect to the videofeed.
 46. The telepresence robot system of claim 40, wherein the requestto create the new tag is made with respect to the plan view map.
 47. Thetelepresence robot system of claim 40, wherein the request to create anew tag is made with respect to the current position of the remotetelepresence robot.
 48. The telepresence robot system of claim 38,wherein the tag information relates to at least one of a position, apath, and a volume, and wherein the control system is configured toexecute an action relative to the at least one of the position, thepath, and the volume.
 49. The telepresence robot system of claim 9,wherein the instructions executable by the processor are furtherconfigured to cause the processor to display the plan view map with anindication of the current position of the telepresence robot on the planview map.
 50. The telepresence robot system of claim 49, wherein theinstructions executable by the processor are further configured to causethe processor to display a rendition of the tag information on the planview map using the tag coordinates.
 51. The telepresence robot system ofclaim 9, wherein the tag information comprises a tag annotation, andwherein displaying a rendition of the tag information comprisesdisplaying a rendition of the tag annotation.
 52. A method for changinga heading of a robot comprising: transmitting a heading to a controlsystem of a robot, the control system of the robot in communication witha drive system configured to move the robot according to driveinstructions; and rotating, via the drive system, an upper portion ofthe robot toward the heading independently from the lower portion of therobot.
 53. A method for controlling a remote telepresence robotcomprising: retrieving at least a portion of a plan view maprepresentative of robot-navigable areas of a robot operating surface;retrieving at least one of a plurality of tags, each of the plurality oftags comprising tag coordinates describing the relative location of thetag and tag information; receiving a video feed from an imaging systemof a remote telepresence robot; receiving positioning informationassociated with a current position of the remote telepresence robot;displaying, via an electronic display, the video feed from the imagingsystem of the remote telepresence robot; displaying, via the electronicdisplay, a rendition of the tag information of the at least one tag onthe video feed using the tag coordinates; and transmitting a command tothe remote telepresence robot.